scale.bythebay.io: Mike Solomon, Strato: Twitter’s Virtual Database Powered by Microservices
all right hello and welcome as she said in Mike Solomon and I work on the strata team at Twitter you can follow me I'm at M saw on Twitter and today we're gonna talk about Twitter's virtual database powered by micro services and we call that strata all right so a micro services powered virtual database sounds pretty fancy and you're right it is but it's really hard to understand why it works the way it does unless you see an example of how you use it it's not like how you use my sequel or Postgres or some other database like that so we're gonna look at it from a bit of a strange angle for talking about a database we're going to build a product feature so as an example we're going to build the tweet off feature I promise I'll tell you what I mean by that very soon but first we're gonna build this feature in a micro-services environment at Twitter and we have lots of micro services hundreds of services and we need to talk to all of them and you run into particular challenges so we'll walk through what building a feature looks like and the challenges you run into along the way then we're gonna do a really quick introduction to Strato cuz that's probably why you're actually here and then we'll actually go through that same process again building the same feature but this time on top of strata that way we can contrast alright so our story begins like very few stories begin with the tweet off so let's talk about this tweet off feature you can see here that we have three tweets of varying quality and we have a simple task we want to determine which of these tweets is the best in data terms we basically want to implement this function we have a set of tweet IDs and we want to return the tweet ID that corresponds to the best tweet so that's pretty straightforward and if you've had time to read these tweets really quick I hope you agree that this tweet is the best I think it's basically just a fact all right so let's get to work let's build this thing and working in micro services so the first thing is let's start a new service so this usually follows a few standard steps first one let's start a brand new empty service this pretty much always entails some service code boilerplate so you're gonna have to set up your main some build files some dependency configuration that sort of thing then we get to move on to even more fun configuration boilerplate we need to set up our deploys we need to set up monitoring dashboards paging alerts that sort of thing there's a lot to do there as well and we also probably need some operational documentation we're gonna need to document the processes and failure modes and a run book or something like that so already we've kind of run into some problems so let's list these difficulties that we've run into so far and later we can go back and see if there's anything that we want to address so already we can kind of see that starting a new service is hard there's a lot of work involved and it'd be nice if maybe there was some way we could avoid doing it it's kind of a problem and once we've built the service it's probably obvious that we have to operate this service more or less forever and if we want to keep this feature around we need the service that powers it and there's kind of no getting around that so let's let's kind of draw what we have now we have the tweet off service trying to build this tweet off feature spun up a new service this is what we have and it contains no business logic whatsoever and of course we still have our list of problems all right so let's come up with a plan for how we're going to do the rest of this so we'll also keep this list of problems around so we don't lose track as we go and we can keep adding stuff to it so let's make what we're building concrete we want to return the best tweets are the best tweet out of those that we have and there are probably already some services that exist to score these tweets so we're not trying to build a machine learning model to tell you which tweet is the funniest or the most impactful or whatever and let's just thing that those already exist we're just trying to build the service that chooses the best tweet so let's assume that these other services exist we're then going to need to call those services because we need the data that they provide and then we're gonna need some business logic so let's just sum up the scores that they return kind of assume that there's some services that score our tweet based on some dimension we're going to add those up and then we can return the tweet with the best score seems pretty straightforward okay so let's actually start implementing this thing inside our empty service shell so let's go off and do that so first we need to figure out who to talk to and by that I mean which services to talk to so that's not always straightforward when we have hundreds or even thousands of different services so let's let's write that down as another problem that we might have this is definitely true at Twitter you're not always sure which data source is the right data source and it can be hard it can be hard to know maybe maybe let's just say we ask somebody and they happen to know so that's what we that's what we do here we probably want to talk to the tweet rating services assuming they exist so let's draw this out again we have our tweet off service no business logic inside of it and we have maybe a couple of different services that score tweets we'll call them that is tweet funny service and the is tweet insightful service and the next thing we need to do now that we know about these services and we'll keep this picture around now that we know what we want to talk to we need to know how to talk to them and that means figuring out their service interface so this can be JSON or thrift or protocol buffers but there's always some kind of schema even if you never write it down or document it and in practice you'll usually run into quite a few inconsistencies here so even with typed interfaces we found that many services support batching or multi get has different names and all services expose errors in some way and we found that these things in particular often differ from service to service so that's kind of a problem so let's write that down as well okay what's this zoom that we've sorted that out we figured out the service interfaces and now we can actually communicate with these services now it's finally time to do what we set out to do which is implement some business logic okay so it might look a little something like this in pseudo Scala we already kind of talked about this we're gonna map over our tweets for each one we call out to some other service to get the funny score and the insightful score will add them up and then we'll pick the one with the highest score and then we'll return that ID so this is also another problem we had to implement some business logic sold write that down like all of our other ones and it said time to light up our service we're pretty much done right almost there's a piece we forgot still but we are in the homestretch so we actually need to define this service interface so we have lots of different choices here we could use protocol buffers or thrift we could use JSON or XML or at least ten others there we have tons of choices here so this is also a problem so I'll write that down and at least at Twitter we often encounter another problem we found that once we pick a protocol we usually pick thrift somebody always wants it and some other protocol usually rest with JSON and sometimes graph QL sometimes something else so we might have to support other protocols as well so let's write that down supporting other protocols okay we're done we successfully built the tweed off service and a micro services world and we accumulated a list of problems along the way great so after all that we have this again this is what we were building we picked the best tweet and what we set out to do was show that my tweets are not the best tweets and it wasn't easy getting here and we ran into lots of problems along the way alright let's finally talk about Strato as I said before strata as a virtual database that's powered by other micro services so let's talk about how Strato models the world like most databases Strato has a centralized catalog of data usually this is per database but here we have a virtual database so in principle it kind of come encompasses all available data but in order to make that true we actually need to expose data via strata so we also have a library of configurable adapters for existing services and also for existing databases and these kind of translate into tables in our virtual database so Strato can talk to existing services and existing databases okay so now we have a central catalog of data that has type and schema information about the data that's available also the location of that data so that it can contact it what protocol that speaks so that it can actually talk to it and that's not a bad start but it's also not really database so let's add a query language we don't really have time to get into details here but we have our own query language that looks kind of like Scala let's just pretend it is Scala but basically it's designed to make it very easy to compose filter and join data together and the main reason that we have this query language is so that we can create a query defined data sets so some database is called these views others call them stored procedures and in stratas case this is how you combine filter and modify other data sets so the data can come from anywhere else in Strato it can come from other query defined data sets it can come from data sets that are powered by existing services by databases that have been adapted to Strato can come from any way so now in this picture we can see that strata let's get data from query defined data sets as well as data sets that are actually powered by existing services and databases and this is great but we are still missing one piece a very important piece clients so my clients I might mean other services that need to read a write data via Strato or even inclined like apps on mobile devices and some of you might see an issue here some clients like servers might want say nicely type thrift API but mobile clients might want a REST API with JSON or a graph QL API or something like that so because Strato knows the type definition of your data we're actually able to generate thrift rest and graph QL api's automatically so that means that your clients get to choose their API protocol and that is a really quick introduction to strata so I know I went through that really fast but we're trying to build a feature here and that means we have deadlines so we have to keep moving so let's get started building our feature again this time around we have strata all right let's start a new empty service I've got our list of problems here let's get this started but wait hold on we don't we don't actually need to do that we don't need to start a new empty service and why don't we need to do that because of query defined datasets instead of building a service we actually just need to write a query that will get hosted by strata so that means right off the bat we can cross off our first problem starting a new service won't be hard because we're not starting a new service and obviously since the service won't exist we won't have to operate it either so we can cross that one off too I realized these claims might sound a little unfounded so far so let's make this more concrete remember the Strato has a centralized catalog of data this tells us the types of the data and where it is all behind a uniform interface and also remember that lots of datasets are already exposed through strata so here we're trying to write a query because instead of exposing raw data like the machine learning models scorp how funny a tweet is we want to pick the best tweet so in other words we want to expose a new synthetic data set once we do that it'll appear as a new data set and the Strato catalog and it'll be just as easy to access as every other data set ok so step one is to find the data sets to talk to but hey guess what they're actually easy to find and read about now because they're in a centralized data catalog so let's cross that out because even though it's still a problem it's now a pretty easy one to solve the next problem is we're going to need to actually call these services and remember this was hard before because batching error handling and other inconsistencies meant that we had to learn each services interface button Strato interfaces are simply the data types batching is handled for you and errors are always exposed the same way so we no longer have to figure out inconsistent interfaces the next step we need to implement some business logic this part actually stays the same I mean I don't know what you want from me it's not in a tie conference we haven't built generally I still gonna have to write your business logic sorry okay let's bring it back to the tweet officer this the tweet off feature rather that we're trying to build so we actually need to write the query so that strata can host it and it's gonna look a lot like before so let's let's bring that business logic back just remember that a lot of the batching and other things are handled for us this isn't really reflected in the code here but that can also be a big deal all right so we have our logic and this talks to some services that are exposed uniformly through Strato and now our picture looks like this hopefully not too surprising okay next step the time to define a service interface so if you recall and Strato all you need to do to do this is a type so you're probably used to doing this in thrift or with some kind of JSON or with protocol buffers but we actually don't need any of that we just need the type and here as we said before this is basically just a function signature so given a set of tweet IDs we're gonna return the tweet ID of the best one and that's pretty much what we need we don't need to do any more work than that to define a service interface alright so let's look at this as a diagram again maybe make it a little easier to follow so we have Strato and we have some clients and we also have an interface between them in this interface is defined by types okay let's keep that around the last piece we need to do is support some protocols and we actually want to support multiple protocols for reasons that we talked about before and in Strato this is pretty easy how is it easy well we can actually derive the api's from the types so we have the type information about every data set this does mean there no custom api's your API will predictably match your data always because you don't get to make any choices about about its shape you can change its shape and that will of course be reflected but this means that data access is always very predictable so in practice in Strato we basically support three protocols thrift rest / json and graft you all so now our picture is complete and clients can choose to talk to strata over their preferred protocol cool um let's see once again we've built what we wanted and this time it was much easier even though my tweets still aren't the best okay so here's the list of problems we've accumulated let's pretend we're in Minority Report rearrange things a little bit and we can maybe look at them like this so here are the problems we encountered when building our tweet offs feature in a micro-services environment and here are the problems that we have in common when building it in a strata environment so let's look at those side-by-side um as you can see a virtual database like strata does not do all of your work for you but it does let you focus on the things you cared about in the first place you to focus on implementing your business logic and defining your interface in this case through a simple type definition and that means we eliminated quite a few steps we don't need to start a new service we don't need to operate that service we don't need to do as much work to figure out which services to talk to we don't need to figure out inconsistent interfaces because they're predictably generated from types and batching is handled for you and errors are always handled uniformly and we don't need to support multiple protocols because once we expose type data it it generates API is automatically based on those types that leaves us with the part that really only people can do currently which is implement business logic and defining what we actually want the interface to look like by choosing types so I hope today that I've convinced you of this a uniform virtual database lets you focus on your problem if I had more time I would tell you a lot more about how Strato lets us trivially apply things like caching and event logging and also apply a simple access control model there are tons more features that Strato brings and you might be able to imagine some of these you can really leverage these uniform interfaces and the power of the types that you have to choose unfortunately I don't have that much more time so I want thanks for listening again I'm Mike Solomon you can follow me if you like bad tweets and M saw on Twitter and if you want to try this stuff out for real and see how it works in detail you can come work at Twitter thanks [Applause] [Music] save and create these virtual sets which are projections of other services but you don't know how many times that's happening right database is consistent so is that is the point unpredictable but is you could be calling a service that's quality services call any service any number of times at that time how many obviously is your right then the number of our pcs that are made is totally hidden from you and you'll probably see it in your latency if that's like very tight then you can obviously go behind the scenes figure out what's calling what and and you know optimize from there another option that's very common is it's very easy to in strata to layer caching on top of things and all the logic is already implemented for you you basically say take the synthetic data set and cache these results you can be read through and right through caching yeah we need to find a we need to find a query defined data set and part of that configuration you can basically just wrap it in caching that doesn't get exposed through the service interface it looks totally uniform to your clients they don't know that it's being cached but it is buying the scenes [Music] so having a global picture of all the services running I'm guessing you can do a lot of global optimizations I don't know if you guys currently do that we have something you're going to do yeah that is something we would like to do in the future right now I've been so busy building the thing that we haven't had time to do that yet but yeah there are some things I think we'd like to do in the future um one one thing that has been nice is because it transparently supports batching we have been able to get pretty good performance overall so it seems that this model is very useful can be applied to almost every power you think it solves everything every one of my dancers is there something this cannot something too things I wouldn't recommend using strata for yeah definitely so right now this isn't totally baked into the model and we hope to sort of maybe like expand this in the future right now a lot of it is fairly key value oriented so there are some use cases where imagine you have like low traffic and what's really important for you is like flexibility and you might be better off talking to a traditional database like my sequel or something it can be tricky to add like new indexes and that sort of thing in many cases yeah thanks the great doc so I might miss the pizza but not well so what makes don't kill such a great interesting thing is this on the growing in Twitter can you talk about the future of QL as you see I'm sure so graph QL in general not probably not everyone's familiar but it's a query language that came out of Facebook that is very product oriented makes it very easy for people building client apps to select the data that they need so the way Strato kind of works with that is we're actually able to generate a graph QL API from that which means that people adding new data are able to expose it very easily it shows up very predictably for clients and then people building apps and so forth they're able to select that data without very much further interaction with the backend there's no need to add custom endpoints for to expose new data for example does that answer your question Thanks [Applause]