Scale By The Bay 2018: Sasha Solomon, 2 Fast 2 Furious: migrating Medium's architecture...
so first off why now why are we doing a migration migrations are notoriously difficult so why why are we doing this so for one our existing front end is kind of old and it hasn't been updated in like five years so it's kind of slowing our front-end developers down we use something called the closure compiler and a lot of people who knew how that worked aren't around anymore and it's becoming a bit cumbersome to add new features also our existing back-end is slowing developers down there's a lot of tech debt and people who used to know how to the whole system worked have moved on and right now people are basically afraid to make changes in it and maybe the biggest thing is poor performance our system like right now is just slow not all of the parts of the system are slow but because of how our system is architected the slow pieces start affecting pieces that would normally be performant so we're deciding to do a migration so what are the goals of this migration one is we want to improve the developer experience want to make it faster we want to make it easier to use and then we also want to improve the performance of medium so this is not really an easy task as I said migrations are aleena Tory ously difficult especially without some sort of migration plan and a unified interface so we want to improve the developer experience and we want to improve the performance of medium and we want to do all of this without hindering product development so that's even harder to do so let's get into how we what our new system would look like let's look at the design of our new system so first off these are some of the things that we just would want in a brand new system we get a design this so let's let's make the best of it so first off we want an easy-to-use client we want our front-end developers to be able to develop much more easily we also wanted some smaller services right now we have a monolith so we want to maybe break up some stuff and make some more performance services and maybe have a unified there's interface for those services and then we want to make sure that our business logic stays within those services we don't want to tell you be like floating around it to different and we wanted to kind of remain there and of course we need to do all of this without hindering product development that's just the hardest thing we can't migrate everything all at once so let's get into how we're going to do this we decided to do it with a phased approach so remember we have two goals of this migration developer productivity and performance so phase one is all about developer productivity so let's kind of look into what this will look like so phase one we want to use graph QL as an interface between our client and our existing API server just a little quick like snippet about graph QL basically if you don't if you don't know what it is it's basically going to allow us to query for all the data we need instead of becoming like hitting a rest endpoint and having the data come back in a big blob so it provides us a lot of cool stuff and so also we want to leverage our existing data description so we actually use protobufs right now we don't actually use them as a protocol buffer but we do use them as a schema so we want to be able to use those things so if we do this this means that we can actually start migrating our system piecemeal and we sort of set ourself up for performance benefits in the second phase of this migration so let's just kind of look at what this would look like so cool we have medium.com it's like brand new and fancy and it's when we hit medium comm we're gonna hit our new react J's client so part of this migration is we want to make the developer productivity better for our front-end developers so where am I going to react yes so our react client is actually going to get all of its data it needs from our new graph QL server and so from this point where does graph QL get its data from we're actually gonna have something sort of this line here's like our new data source and anything behind that line is like where we're going to get our data and that's where our old API server is going to live so right now how this works is we sort of have all of our business logic that's staying in the backend and like I mentioned we have protobufs already we're not using as a protocol but we do have a schema and so we're actually able to derive our graph QL schema from these protobufs so that's pretty cool we also are able to like harness the power of graph QL and deal with any discrepancies between our proto's and what the client wants because we'll have this graph QL server here and what's extra cool is so now we have our API servers still talking to the same databases but again this is like behind our data source and with graph QL we're able to sort of compile all of the stuff that we're getting from our API server it's all gonna be populate all these objects and everything will be populated in our graph QL server so we have this like cool line here so what's cool about this is that our new system is sort of being built alongside our old system this API server is our old system and it's you know we can still hit different pages using our old system we now we have new surfaces that get migrated using our new system because it actually uses our old API server so cool we're migrating without hindering product development so that's awesome we're actually able to migrate a bunch of this stuff and still give product engineers the flexibility to begin working with our tools sooner and provide value as soon as possible ok so we did that now it's on to phase 2 which is performance so what are some of the things you want to do in this phase we want to be able to break out break our old REST API server into services so basically start chipping away at our old API server turning them into services not necessarily micro services here but basically just services that make sense maybe some parts were less performant and we can break them off and kind of you know dedicate resources to them and leave other things there this also means that we can start retiring our legacy systems once we start breaking apart all these old pieces we can finally get rid of the pieces that we're making things slow and this is all all in the name of performance we can also make our endpoints more lightweight that are designed for a graph QL traffic so basically this this phase 2 is all about performance so let's look look at what we get in phase 2 so this is what I just showed you this is you know kind of the first phase where we get the developer productivity and our API server is our old API server so phase two starts to look something like this so our graph QL server is still using our old API server for whatever we need but then we start breaking off pieces of our old system turning them into services and because we have this unified interface that means that we can use graph QL for that that unified interface and it makes things a lot easier to migrate so let's kind of like look more into this so yeah so we have our services here we can break out and optimize these and we just basically gain way more in the way of performance and like I mentioned we have protobufs and graph QL is already using them so this kind of means that we could possibly use something like G RPC which uses protocol buffers as and we can actually use that for interest service communication so that we buy a lot from that as well so let's also look at our graph QL server like I mentioned this means that because sorry because this is a unified interface that means that we can actually use these protocol buffers from our services that we can use there and then basically it games us like a ton of stuff in terms of performance cool so eventually we get to something like this cool so over time we can start breaking apart our old system into a bunch of services and eventually we get to something where we've actually gotten rid of hopefully our old system that will probably still exist in some way or another but we've we have this new unified interface that makes it a lot nicer so what do we have here we have our clean intuitive client we have our really awesome modular performant services and we have this unified interface graph QL to tie them all together so maybe you can tell but a lot of this migration wouldn't have been possible without using graph QL so let's talk about how the structure of our graph QL server actually helped our migration cool so when designing our graph QL server we wanted we wanted three three main things and we want a clear separation of concerns and keep in mind here we're dealing with like legacy data because we're trying to migrate an old system so we have a lot of legacy protobufs they're not always what the client wants so we have to keep that in mind so one it should be easy to alter the shape of the data legacy data coming in shouldn't affect what the client wants - it should be clear what data is for the client and what data is like raw data and then it should also be easy to add new data sources it shouldn't be confusing about how to do that and where the data is coming from cool so we've done this with this layer cake of responsibilities such as repositories and our schema so I'll show you what this looks like and I'll briefly go through this but then we'll dig into what each of these layers is doing so say we we get you know we hit some our endpoint and then we get a bunch of data back it looks like this and then it's not quite the shape we need so we're gonna take it through these different layers and kind of shape the data in our fetchers it goes through to our repos wolf will shape it a little bit more and then eventually it'll go through our graph QL schema and it'll be actually what the client wants the this whole graph QL server is sort of you know for the client like graph QL is all about what does your client want and making sure that they're getting the data that they actually need not that the the data that you just like happen to send back to them so this is all for the client so let's kind of dig into exactly what each of these pieces is doing so first off fetchers are for fetching data from data sources so right here basically each fetcher is going to correspond to a rest or a G RPC endpoint and data that is fetched from these endpoints should have already gone through any business logic or changes so no business logic is happening in your graphical server so also they should require it should correspond to an endpoint and also require a proto buff so basically like I talked about we use protobufs as sort of like a schema so each of our fetchers corresponds to a protobuf cool so in this step sort of what's happening is we get something back from our fetcher it's it's corresponding to an endpoint and this gonna go through any business logic has already happened and this part is sort of like our protobuf really like going through and it's sort of changing the data to be matching our protobuf schema and then it'll go on to the next step so let's look at what this actually looks like in code so this is Scala if you're not familiar with Scala I promise it's not too scary but hopefully you know about Scala cool hopefully you can see this so this is our factory trait and all all of our fetchers are going to conform to this trait and you can see that we have a get method here and this is kind of what we're going to use so pretty pretty simple so this is an example of a fetcher like an actual implementation of one it's our post fetcher this is our medium client so this is like our rest server this is like where we're going to get our data from and here as in the previous slide this string here is gonna be our ID and the post is our protobuf post so that's the schema it's conforming to and then we just get our object and then the you can see the endpoint that we're getting it from so cool and that's pretty much it for fetchers so this is the next step repositories so repos are what the graph QL schemas will use as a data representation so this step is is kind of like the most important step and it's like the the coolest step for doing a migration so in this step we're basically shaping the data we're cleaning up data from our data sources and we're storing it in our repos so in this in this part of our graph QL server we can basically like hoist up feels like move them around and change the data shape to be what the client actually wants and again we don't do business logic here there's no business logic in our graph QL server and also our repos we'll use our fetchers so they they won't actually fetch the data themselves they'll use the fetchers to fetch the data so the repos are actually like creating the data shape we want but they don't know where the data came from in that sense so repositories are for data shaping so let's look at what this looks like in code so it's going to look very similar to our fetchers we have our trait here and we're actually defining a multi get here and we'll do it in the next step but an hour.get is defined in terms of a multi get so these just come automatically when you define it this way so again like all repos will conform to this tree and this repo will only know about factories so let's look at what this would look like if we implemented it so this is like a post repo and it's gonna take a post fetcher and then again we have like the ID which is going to be a string type and then this post here is actually going to be our our repo post so this is where we're actually able to do some data shaping so this is where we could add like new fields or move stuff around and so when we do this multi get we're actually gonna map over it and use this from a fetcher method and that's what's going to take our our protobuf schema post and turn it into our bure depository post cool so again we're taking the raw data and are transforming it into the data that we need so this is the final step the graph QL schema and it's gonna take the form of what the client needs so this is the final step so the graphical schema is only going to use data from our repos and will not access factors directly so this is the graphical schema doesn't know about fetchers at all or where the data comes from it only knows about repos and all of our schemas are derived from repo objects so this means that there's no confusion about like what the data shape is or where it's okay to manipulate the shape of the data because that can't happen in our graph QL schema the data in the graphical schema is always derived and it's derived from our repository object so repositories are where we're doing data shaping graphical schema no none of that so let's look at what this will look like in graph QL we're using sangria because this is Scala so this may or may not look familiar if you've used graph QL this is pretty simple but we're just constructing our schema here and we're making a query and it's going to be for a post so you can kind of see we we've defined some some things like medium ID this is just an ID type and then we have things like our medium query we're calling it query and we have a field on it called post and this option type post this is actually our graph QL post and it's something that we've derived and then we're just going to resolve it by calling out to our repo and passing in the ID so this is like fairly simple so let's look at what our post looks like super simple like I said we're just deriving it from our repository so in sangria there's this thing called derive and basically it's just going to take the data shape that's a case class from our post and just turn it into this so it kind of automatically does it for you so that's all we need so in this in this piece of code this like post option type post it's just this and this is all you have to do so that's pretty awesome so in the end this is kind of what we get so again we're hitting our rest endpoint or a G RPC endpoint and it's going to go through this this layer which is our protobuf kind of shapes the data a little more to be what our client wants then it'll go through the repositories and that's where we do all of our data shaping into our repository object and then it's going to go through to our graph QL schema where it's finally what the client wants and all of these layers are distinct and it means that like we we don't have to worry about what the different steps are they're all distinct and so there's not going to be any confusion about where data is changing or what's going on cool so as we all know migrating of system is hard but what's cool is that graph QL has made migrations a lot easier we were able to design a system that was migration friendly we were able to migrate off an old system and we were able to do it without hindering product development and so that meant that we were able to basically gain the the benefits of the new system along the way and I think that's pretty cool thanks everyone I don't know if we're doing questions but I can take questions or you can talk to me after yeah you sir in the back [Music] right migration back can you say that that last bit again so the question is like do we oughta me are our data shaping stuff from our protobufs during this migration is that the question so the we don't the schema so the the graphical schema is derived from the repository section so the protobufs don't change so the protobufs just stay the way they are because basically like they could be they're like old and we don't want to like go through like messing with them so those stay the same and then all of the the shaping of it is happening in our repository section and that part is an automatic that's by hand right now so there's like a library that we use that'll take a proto buff and turn it into like basically escala keys class and then from that we kind of change it around but that's all manual so the question is do we have like one giant graph QL schema and like you know how we're growing is it like getting bigger or is it just like broken up yeah so right now we only have one graph QL schema just like one big schema file um it hasn't gotten big enough that we've needed to break it up but that's like something we plan on doing sorry is your is your question like do we have problems like taking different parts of schemas yeah so I guess your question is like is there any problems like making changes to the schema we haven't had any problems with that most of that stuff is taken care of just in like when you're making PRS and like merging so we haven't had any like schema conflicts we don't decide like I we make changes to this schema like in our graph QL server but it's sort of like that happened how that happens is like we make it in the graph Kiel server and then the client just has to like match it so we we don't make like backwards and compatible changes so like like the according to like the graph QL spec when you make changes like that it's like preferred that you make like backwards compatible changes so like you'd have to change your schema and you can't like if you change something like I'm like kind of when it's running I guess like that won't work so that would break everything but basically that's just like a best practice thing you mentioned that you wanted to do the migration without slowing down product development what is kind of your decision criteria for what you're grading into a new service your services or your chipping away um can you can you say that just like a little bit louder for me sorry so you you under my grave without slowing down product development so how are you deciding what to migrate out of your model life okay yeah so the question is how are we deciding what to migrate out of our monolith how that how that decision is made so sort of it's starting out at first basically it was stuff that wasn't too like not too important or just like not as big of a deal so that you can kind of like see how this is working and then kind of as that goes continuing to do that as far as like services and stuff basically the ones that make the most sense like authentication is like an example there's also something so like medium is like you know our whole thing is posts you write stuff a medium so that's like a big big hunk of our code base and if there's just a lot of very specific code about like creating a post and validating posts so that's like a good candidate for a service but other things like it might not make sense I think splitting up services into too small of things might isn't always like the best but it kind of is like a case-by-case basis yeah well if there aren't any more questions I think that's it thank you all [Applause]