SBTB FinagleCon 2015: Yongsheng Wu, Finagle at Pinterest

Recording: SBTB FinagleCon 2015: Yongsheng Wu, Finagle at Pinterest

I really appreciate that have its ability to talk to the community pinterest i joined about three years ago we've been using finna go about two and over two and a half a year now and its handling millions of QPS for us so I'm justice i'm going to really quickly go through like you know what interest is Pinterest is really solving a discovery problem what doesn't mean is that like if you know exactly what you're looking for you you go to Google you tapped in and Google typically give you you know something you know you exactly are looking for but you know for questions like you know what I'm going to have for my dinner or like you know what kind of our pieces I'm a like and that kind of question you go to Google Google may not give you a satisfactory answer and we build a service that try to help you discover things you like we do that by you know you can go get on to our services and follow users and boards and you get a home feed it's kind of similar to what do you get from facebook and you sort of a friends people or you get to Tori follow people and but we also recommend stuff to you based on the stuff you've interacted in the past and we also make interests really the first class citizen you can they're the interest are really clusters and then they're related to each other and we exposed to you the related interest and then you can navigate through this and really find stuff you like and any problem so when you solve those problems as skill it becomes much harder and we got 50 billion pins and and and we got 1,000,000,000 boards and people put all those pins and supports and that they actually categorize all the stuff for us and we get much much cleaner data and that's sort of you know with good data it it comes really good you know your machine learning and all that stuff works whole lot better and we're roughly doubling our data set and you know every six months now I just want to talk a little bit about you know the the service oriented architecture transition we're going through and then why we pick you know finna go as a framework to do such a transition and so when you guys start it sorry so Oh too fast okay so we get started really you know back in 2010 we got three founders and one developer wine engineer and fast forward to you know December last year's is this is lat last time we have a hands kind of photo nowadays we have about you know more than 600 employees and to surrender engineers and you know if if you're on a mullah listicle kind of code base you know you can imagine that people going to step onto each other's to a lot and then service-oriented architecture you know by breaking up this model is a good base into smarter services and really a lot of good things happen one is really you know the ownership and also long-term ownership the you know the service owners they own a services you know from its inception and maintain it and really knows the context and working really hard to optimize it and reduce operational overhead because they own it and another thing is that like it really you know the services talk to each other and based on some API some contract as long as the contract stays the same you know with Xion you have sort of the flexibility to change anything but if you're not meeting your SLA and people going to they also know who to talk to so there's a lot of really good things happening with service-oriented architecture also but but of course you know there's a lot of more complex that he comes with a service-oriented architecture because you know martes went through that you know in a lot of details but that what really that means that if you're going to serve as our interpreter with hundreds on you know then you know 20 30 services in your stack I you really need a really really good service firm work and and and capabilities to distribute it debugging and tracing and really knows what happens across your stack so wife in a go and then like you know everybody probably all read the real q manifesto and you know I'm just going to keep this short because i only have 20 minutes but my promise to you is that if you use finna go and if you use in right away you get all this in your service and then the other thing is like when we get started we're really just a Python shop and it's a django app you know as as the services you know gain popularity we have a hard time in scanning to to me the load from our users and because though also because the way how we use python we use python is sort of a single process a single thread way and then if you can imagine that let us say you have a server and the 100 requests coming in and every sir every server now pick up warm Pleasant deal with it and if you're back in somehow you know that your database is slow and that's it you can only handle 100 k vs but it's what it really needs is that you can you need the capability to handle all those work concurrently but that means that you need to do the logging and all that you know stuff correctly and for a code base that was not build with that in mind it's actually really hard to do and but lately we introduced something called Jeevan into our stack sort of partially solved the problem but just to be honest like you know python compared with the JVM based language like Java or Scala it's it's much slower and we really liked the future program my motto it's really is so much better compared with xenos callbacks or like you know javis native way of dealing with future also finagle comes with visibility into like what's going on in your server and it has awesome community and like you post questions to the user group typically you know I've never waited longer than 24 hours to get somebody to respond to me and so you know as I said we we come from Melissa go serve really big code base we break them up into services and along the way sort of this is a typical sort of best practice we follow so that you know we did all that stuff without really introduced you know huge eval BD events and the availability events we had really is like you know either dynamical service dear discovery has issues or your you know DNS or the back that sort of the things typically caused huge outages and with vinegar itself as always has been robust and typically it's us who did something wrong now that we're on so first thing is that like when you do this type of migration you want to avoid a change of functionalities the reason why is because typically you wanted to sort of be able to do some sort of comparison to making sure otherwise if you change the functionality what do you compare to it makes things much harder and you wanted you know use test-driven development if you have sort of a really complicated logic because you know if you let's say you go do a mass long kind of development and then you're like okay let's try with real traffic and so that's sort of very not not so responsible and also at the same time if you if things not going as you expected your like you know where where is the problem so typically what do we do is that we play with duck traffic and then dr. Foote you know which means that you sort of replay at the same time as the real traffic community goes to into your oath system and you have a percentage of the traffic goes to your new system and then of course there's a complicate shin comes was like what how do you handle right traffic right and and and you know sometimes you have some like separated data store somewhere else and then you compare the two and one way or we've been dealing with this is that like we replay the traffic based on the user instead of just you know random if it involves rights and so that then you can pick that right that the data belong to that user through the comparison another thing you need to pay attention to is that when you do this sort of you know dark traffic kind of mode and you're really causing additional loading into your system because you know you're not only handling your live traffic you handed me you know your your new system is doing some duck reads or whatever and that's additional load two-tier system making sure you have that kind of capacity in place another thing is that it really helped us is that our capability of being able to sort of dial up and down the amount of traffic we play through the new system so that if we find something wrong whatever we can download button down and the same thing applies to when you really everything is ready you're ready to like you know really have have your new system handling live traffic it's the same thing if they're all of a sudden find something's wrong it'll be a nice that you can like real-time dynamically like switch the traffic back to your own system and just sort of show like you know I mean it's at this point is impossible for me to actually put every single services in your stack int into one graph I sort of just picked you know you know if you could get onto our service and create a pin this is you know sort of all the services that's involved to do that I was planning to cover some of the interesting services we have but like I'm you know I only have 20 minutes so I'll skip that part we have some people which they're going to hang around here and if you are curious about some of the stuff interesting services we have and you can ask them and also encourage you to go to our engineering blog not and there's a lot of interesting stuff there so a proper service proxy so we we have a proxy and we plug into all of our Java finagle server as I said you know finagle give you a lot of disability already out of box like you know into what a service is doing but in sort of a distributed system you want to know like if you know you know let's say you you all this doesn't have a loti show your services handling much more than it's capable off like what did you do and because your service there could be like 20 different clients are the client cluster calling this a particular service cluster and one thing is really useful that you want to find out who's the bad apple who is causing load to your system and then you need to differentiate the traffic based on the client or like sometimes like a particular API endpoint is heated up because somebody has a bug and you want to figure out you know which API endpoint is causing the problem for this type of stuff you know out of boxee you don't have that kind of visibility what I heard in the morning is that seems like a turd is building this thing into Scrooge and you know once that I'm sure ties that and and maybe the world are going to be a whole lot better for everybody using finagle but if you wanted to do the same thing as with it it's actually not that difficult so we use something we basically use Java dynamic proxy and this is a way to deuce code injection if you have interface and implementation then like it's easy to use that English porosity to inject some dependency and then basically we hook into our casino logic to do request logging to keep track of like a particular request from which client is coming from and we record latency metrics based on sort of / kind / API endpoint kind of purp and I'm going to show you a little bit like what our request log is like so you notice the first field is kind of like you know from which client and cluster the request is coming from and second thing is I from which client and server is coming from and also it records like which API endpoint is calling with what parameter and in this particular case it's a fair request what an exception you're getting and then you can we actually hook this up into a real-time sort of a data pipeline we aggregate you know long multiple different dimensions for example from which client requests coming from from you know which server the request of the exceptions being strong because sometimes you have like a 100 servers in your server cluster and then one of the one of the guy you know let's say house you know something bad going on some other processes running wild or whatever happens with it it just swim or exception than the others and then you can easily figure that out and you know maybe there's API endpoint it's bad and everything else is fine and like sometimes the exception type will tell you information about like what's going on in your system so you have this information at your proposal at real time and then it comes really handy if the if you have production issues so the metrics so this is this is what I was talking about is you know we require metrics that perc line / AP I and client level so the first line you can see that it's a you know the basically QPS or request can't tracking from a particular client the second line matrix is really for a particular API endpoint from you know which from a particular client cluster and then the third one is really you know poor API endpoint the distribution of the latency so you have more detailed it sort of information about it and then I'm just as we're going to very quickly go through like a red limiting stuff as well basically you know you want to protect your service I say you're you're kind overloading you and you need a way to shut up shut off traffic and then this is also building into our proxy because that this love as I said we have visibility into which client is sending us a request and you know as long as you define some kind of policy was real limiting then you can just reject the requests coming from the client I mean like we have like basically a real-time way to dynamically change the the red limiting so dynamic services Gregory the one of the key thing is that you want the capability to be able to quickly increase the size of a class or decrease your class very if like there's not much load going on and then unfortunately this is also one part of the system that could cause you cause you huge huge Japan and the way how we did it initially was not different that different from you know a lot of other people which is that you know you have your server up and then the server created some kind of ephemeral noting zookeeper and that sort of become your group membership and then you're like a client base is give you subscribe to it and knows like who's alive was dead and and all that good thing and you can dynamically manage your classroom um that way but the things that you know we have this happen multiple times in our stack which is that you know your zookeeper now becomes you know this is a tier 0 component it has to be always upheld always healthy and and sometimes actually on fire when when it's on fire and then like okay so everything all the best you has it off so what do we did is actually we have the we basically have the Guru membership a class for information stored in the local file and we have a way to say you know if there's a massive difference from the google membership will get from from zookeeper than just rejected and so this way if you know we keep a local copy on the disk so and and that typically and you know after we put this system in place we haven't have any like big issues with it as I said you know gaining visibility into your entire stack is very very important in service oriented architecture and then like but it's very typical for all the organization to have you know multiple different languages multiple different kind of platforms in place it's like not typical to say you know we use a fini go to finna go where everybody has a right job alright so here I just give sort of example we basically just make every single platform sort of that's our goal eventually right now only have it for Python make it understand finagle server protocol so the way how finagle was being able to do this or the tracing and all that good stuff is really coming from the power of having a header like HTTP has a header but Finn a but if it doesn't have a header out of the box and so basically you know tear gas build this into the protocol the way how it works is through some a handshake vesak alee sending a magical request from the client to the server and in this you know this name is the subject to change but like you know it is illustrated here and if the server aniston's finna go protocol respond was you know upgraded reply and from that point on kinda will just you know send the header along with the payload and when it's getting the response back is that being a response header as well and if it doesn't understand it in like Python it would translate 80 application exception and then the kind basically knows all the servers and doesn't understand finna go particle and then just from that point on just plain vanilla so you can basically you know build the same thing if you have other sort of language platform or firmware using and because making you know throughout the whole stack and speaking the same particle using same kind of convention you know is very important because otherwise out that you can can really do much so I have something else but like running out of time but basically if you have a tool to allow people to generate some some style code for for services and it'll boost / as a developer productivity a lot that's pretty much my talk and now open up for questions yeah I apologize that I prepared a much longer burning of the talk and I didn't realize the 20 minutes so it's rush rush through a whole lot of things and but if you're interested in knowing more details you know feel free to shoot me email or like in our you know write me a message on Twitter or whatever I'm pushing about I have a question about songs discovering why don't you use to keep those observable no instead of actually coming it into a local farm oh so the problem is that that's exactly what we did initially but like that basically require you to have zookeeper money and he'll see all the time and if if zookeepers in trouble then you know the whole entire stack is in trouble and that's exactly what Ryan 22 years ago labor day weekend I was live drive to LA my manager was calling me and it was the seven our heritage if you were really doing all over in what would you do if I'm doing so so actually lightly I've been giving a lot of thinking I'll on this is like I think it should be in general a general pattern is really uses recuperar as the coordination system and have like another system sort of playing then in the middle and this applies to service discovery it also applies to if you let's say you build a sharp manager a kind of system and like you know that's kind of the same way to do this so have those those guys have up to real-time up-to-date information and have those guys relay the information to to the others and the 1i oh say I just going to add very quickly at a detail there the first time we run into this kind of problem we add a whole bunch of observers into our system so the thinking is that the observer can help scale out the reeds but the problem with zookeeper is that was out there certain patch when any kind of connecting to observe reason the they need to ask them in the session as soon as a session requires the master knows that you know the coordination and and and it doesn't it didn't work basically you