Scale By The Bay 2020: Brecht De Rooms, How Micro-Service Patterns Change When the Database Scalable
[Music] thank you i'm super excited to be here at gale by the bay and i would actually be even more excited if i could see it in real life but that's not possible at the moment uh so let's dive right in um so i'm here to talk i'm i'm braked i work at fauna which is a global database i'm here to talk about microservice patterns today and more specifically about microservice patterns and how they might change with scalable data apis so the focus is really on scalable data apis here today before we dive in it's always good idea to frame the discussion in history and if we look at the evolution of data in the 70s we had traditional databases right after not right after 30 years later in 2000 we had the nosql movement and they promised scale but at the price of consistency which means that if you write data and you read it at multiple locations in the world for example are in multiple nodes then some of these will not get the correct answer directly so that's eventual consistency and in that case it might also be the application that is reading uh which might make it very very hard to write your business logic correctly and they also did not provide relations so it was also typically missing in nosql at that time we had an acceleration of technology and although technology was accelerating significantly we had databases popping up like mushrooms it actually took quite a while to still to actually come to the point where we have globally distributed databases that scale yet also provide consistency and the same consistency guarantees as you would find in the traditional database and provide relations in 2012 we had spanner and a lot of people actually call that the new sequel uh movement or something like that and two years later we had fauna which uh also provides the same guarantees as spanner only it doesn't prov need a specific hardware while spedder requires highly synchronized clocks fauna can run on any hardware some might call it nosql 2.0 i would not adopt that term because it's already a bit overloaded today and many people are still associated with no consistency no relations although foreign schema switch is also often one of the nosql properties why did it take so long and that's a question with a very simple answer well it's probably the most difficult problem to solve and if we actually put a parallel timeline on top of that and we look at when software service oriented architectures and microservices became popular and when these patterns were developed then we are actually more or less in the microser in the nosql and traditional databases era that means that most of these patterns were developed with the with the dot in mind that you only had no sql which skills but doesn't have consistency and doesn't have relations or you have traditional databases which might not scale today we are here where such databases do exist and we can also say the traditional databases are more and more scalable today and the question then becomes how do these patterns change when the data is actually the data layer has evolved so significantly to answer that question the first thing we have to do is why microservices and what motivated us to actually start with microservices and many of us here uh probably and let me quickly open the room so i can see whether questions come in and many here will probably [Music] have the same answers as as i had when i asked okay why should i use microservices and those were this layer and average skills separation of concerns scaling in general freedom of choice ease of maintenance and it actually looked very appealing and many also turned right back and found a new found love for monoliths when they notice that these do introduce complex patterns operational overheads and eventual consistency and it's important since we just looked at the history that eventual consistency was something that was considered for a long time a necessary evil and these patterns were often developed in in context of that um today we can make a more calculated choice we have seen a lot of companies try microservices we have seen them fail we have seen them succeed as well so we can make a calculated choice but these choices still remain super hard to make and that is because we miss details often we see something that will solve all our problems which looks like a unicorn and if we then start developing we see the dragons appear it's not our fault by the way that's good marketing because they're just doing their job and highlighting the positive things and the other thing is that choosing between monoliths and microservices is actually a combination of many factors there are lots of pros lots of cons some what heavier than the others and what doesn't make it easier is if you choose a specific data layer it can have a huge impact on their choice it could skill tip the scale towards monoliths but it could also tip the skill toward microservices because some databases actually offer specific features that might make it easier to adopt microservices or it could even make a major difference when you choose between a shared database or a database per service so it could have an impact on the patterns you choose one thing we can probably rely on is that when it comes to microservices and this is actually super interesting because there was a panel about microservices so i really would love to hear your thoughts uh but one thing we can probably rely on is that for processing it's a no-brainer microservices will help you scale up processing when it comes to synchronization and data persistence then it becomes very difficult so if we look into this with a scalable data api we first have to list a few reasons why microservices so we can put them next to our discussion so one of the things that we will look into is a separation of concerns and teams independent scalability of components which is of course uh more on the processing side but very related to the first one ease of maintenance is often uh used as well as an advantage and that is probably because the code remains much smaller per microservice freedom of choice if you use microservices you are probably you are often free to choose your language or your database specifically for the microservice so specific database features that are very well suited for a specific microservice might help you a lot there and then there's the reason that the database layer never scales or rarely scales and that's again we can already bar because we're going with the assumption that you're adopting a scalable data api but then you might also wonder already if we choose one specific scalable database do we still have the freedom of choice and we'll address that later so imagine we start off with a monolith and we which is often a good idea to actually start with the model and then start splitting up microservices as you need them instead of starting microservices immediately and we detect a bottleneck or a desire to split of something and in this case it might be the user management for example so we'll make a user's microservice and we'll call the rest other service the rest service is pretty confusing so let's call it the other service immediately when you split off one microservice you have data choices for example you could use separate databases per service for schema for instance imagine we take per service then the rest would have one database and the user's service would also have one database the advantages are that you can choose the best database for the job it's loosely coupled however there are big guns as well so cross service joins if needed become a serious challenge you might have to manage the complexity of managing multiple databases and you will be you will need to be able to split data but that's often the case that you can split data that's usually pretty simple but the biggest one in there is probably that you just introduced eventual consistency if you need something from both databases need to write about databases at the same time yeah there's something you can't do unless you implement that on top of it with sagas or things like that which will dive in but the problem is that we would then move this complexity that is also already solved by database back to the engineer the other approach and before i go there if you're interested in the the pros and the cons of these approaches a great great resource for this that you probably already know is microservices.org you can find a lot of information there if you want to go dig deeper um so the other approach could be one database we could hook all services on one database in that case it would be uh asset so whatever transaction we do it's a one database so we could actually benefit from from the asset properties of the database it would be simpler to operate but it introduces a lot of bad things as well schema coupling time is coupling a lot of other coupling that i didn't mention here and you might say okay databases need to be scaled now um and can i still use the right database for the job i do realize that this pattern is often considered evil for many people and i'm not here to advocate for one of the one or the other i just i'm here to show what actually changes when the database layer scales the first thing we can obviously already change here if we scalable data api if i this and we would for example use fauna which is what i'm going to use because i know it best databases need to be skilled yeah we can remove that one because we're assuming the database skills here it's not your concern anymore simpler to operate well that's an argument that is here but that is often used but actually this database is indeed zero operations it's one click to get a database but it's also true at the other side one if you would use fauna for both and i assume if you can't use fauna for both you would have specific requirements that there are other databases that make it very very easy to set up a database today in fallout for example you could actually simply make databases in databases and you may can make as many databases as you want so i would say that zero operations are the fact that you that it's harder for microservices to maintain these databases is not really of a strong argument anymore anymore today so let's remove that but let's look at those two schema coupling and time based coupling because those are the reason why using one database is often evil however if we look at it why we need that we typically need that so that two teams can work completely independently without having an impact uh on each other that means that you could actually parallelize development which is something that a lot of managers would love to hear and that makes sense right if you can parallelize development you can scale up teams more easily but if we look at the actual thing that we are trying to accomplish here then it's actually logical boundaries that we're trying to make and maybe that microservice pattern of splitting up databases is not necessary here it might be a good idea it might be but in this case we could for example use another approach let's say we use user defined functions which is much like a stored procedure only you would implement it in fauna in the same query language as you would normally query and you would say okay but user a stored procedure user defined function yeah you can make an api so that one team gets an api to the other service and you won't impact anything as long as you keep the api the same but what about migrations but there are also solutions for these uh so if we combine the user-defined function and look at how fql works which is the fauna query language this is how we would create a function and this is actually also pure fql so the f-code is also used to create a function uh we could easily split off the logic that we're going to place in that function imagine that we have some other fql that is the logic and this is actually pure javascript um and basically what i'm showing here is that i've got a super composable which is very very useful for what i'm going to show of an approach to actually separate the apis if we then combine that with the security system we could say we have one rule that says this team can call this function while we could be making another role that says this team can only update that function and we could integrate it in the deploy cycle with a specific key that only can update a function and the other key can only call it if we then go further and look at the fact that fauna also has temporality we could also tackle migrations for example we would just say okay the first logic is some fql query second logic is the other fql query for version two with this we will of course deliver the same data and it's important to notice that funnel is schema less so you don't have schema migrations so of course the schema does leave something somewhere and in this case that would be in the code of course here in the query then we could have migration logic again in a separate statement this could even be in different files essentially what we're doing here is function composition and then we could actually get to our final query and the query would simply say okay we get a user because we're going to do something with the user in the microservice um then we get a timestamp and because of the temporality of fauna everything has a timestamp you can even go back in time if you want would want that and then the only thing we have to do and this is where fql is very strong at like conditional logic we could say if the timestamp is smaller than now then we're going to execute logic v1 and immediately we're going to migrate the the data for example because the logic for v2 requires a small change to the data and the next time you're going to call this on the same document the timestamp will be updated so you're going to automatically call logic v2 that's where you could actually easily make a separation is that perfect of course not so let's put a neutral smiley there but what if you could actually execute transactions over multiple databases that would be awesome right however i have to disappoint you it's actually i mentioned it because it's feasible but we haven't implemented it yet the idea here is that it's time for databases to adopt adapt to your servers and microservices needs instead of the other way around instead of creating algorithms uh to cope with the the things that it can't do um if such a feature like transactions over multiple databases would be of interest uh feel free to open a feature request i don't think we have one already although it's an idea that lives in uh internally then we have time based coupling of course what happens if you write to the same data what if you have a noisy neighbor what if you have locks we can already eliminate noise the neighbor because that becomes the problem with the scalable data api that shouldn't be your problem to worry about if you look at fauna for the rest for the writing to the same data unlocks this is inspired by kelvin which actually does optimistic calculations uh we have written a whole article on css tricks if that interests you this image also comes from there you might say okay but optimistic calculations those are just like lox rights and therefore still locks and that's true but reads are not affected by these optimistic locks which is not the case for other databases with occ checks and at the same time for the rights there are a number of optimization techniques that you can do at the database level especially with such an algorithm these are more questions for our engineers if you do want to know more about that join our community but what is important as well is that besides of the optimizations you can do on the database level you can also do a lot on the application level imagine that you have a database a collection and a document and imagine you have a bottleneck in the document because many people are writing to it at that point you might argue that this is actually an application requirement if you are able to split up the documents and therefore able to split up your data and put it in different microservices then basically you don't have that application requirement and you could have splitted documents as well uh in the in one database so there are always workarounds imagine you have a counter for example you could even say i split up that counter in multiple counters i randomly select a counter where i would increment it and then would aggregate it at the moment i read however the last disadvantage here use the right database for the job it's true that no single database is good for everything however if you look at the database worlds like the different types of databases oltp relational document stores search engines all up graph databases what we're seeing the last years is that databases are growing more together the gap between databases is changing we have much more databases who are multi-modal and if you look at fauna for example it sits around there it's relational it's document based it has search capabilities have graph like capabilities is scalable and has temporality built in and of course it's an oltp database however i repeat it's still true that no single database is good for everything and there i would argue that it typically boils down to data requirements do you need consistency do you need a global database do you need transactions or do you need disaster recovery and depending on those you would actually be able to choose then and if you take that into account that it depends on the data or the data requirements maybe this makes more sense that we have a set of services that would actually integrate on the database level which actually have consistent and transactional data needs in theory it would be possible if your data layer is scalable and then we have other specific data needs where we would actually have a database per service for example in case there is a database we really need for a specific service but it doesn't scale we could always take a different approach there or actually combine the two if a service needs data from both an example if we would only combine two databases and we're taking two scalable databases here fauna and rocksets we would have all these uh possibilities already so we would have databases relational documents graph-like temporal and is no ltp database where we could actually do the bulk of our data operations that needs consistency and need our data to be safe and then we could use the temporality features for example to pull out change sets uh and actually stream the rest to rocksets which brings you with real-time analytics search engines and olap so we have looked at one side in the other side uh if we would have separated database per service we would have the example the advantage of loose coupling but we have the disadvantages that cross service joints is a challenge so we did that because we want our teams to scale independently even there a database or specific database features could help a lot so what if service one needs data from service to there are multiple ways you could actually make that happen even if you have a database per service you could have transactions in the microservice you could merge a microservice again but yeah that doesn't make sense in this case because we just split it off we could have asynchronous messaging or we could actually have a materialized view so the second service could have a materialized view on the data from the first service i would already borrow this one because we split it off we don't want to merge it again then transactions in the microservices that's very strange at the idea it's uh very similar to the saga pattern but in essence what you're doing is placing the problem which has been solved by the database again in the engineer's hands which means you're going to spend a lot of time on something that databases have spent years and years to try to solve for you of course these two are still a very very in my opinion because this is an opinion of course are still very good options [Music] and let's go back to the same idea as before it's time for database to adapt to your servers and microservices needs and for these two that means fauna could for example help you with your with the temporality uh that means it enables change data capture it enables the easy implementation of a queue you could do seeker cqrs without actually implementing setting up a different queue infrastructure i mean and so we even have push-based streaming which would actually get real-time uh data into another system in a push-based fashion so we could say that cross-service joins is a challenge but it can be we can get a lot of help from the database there as well so let's put a neutral phase there and once we then put these two together i would argue that it typically boils to the database it typically the choice between microservices and how you would implement it whether you would stick with a monolith whether you would go to microservices or how you would uh which pattern you would choose if you go to microservices purely depends on what data requirements you have are you able to split up your data perfect go for it do you accept eventual consistency that's already something you have to think through because often the issues with eventual consistency are only apparent after your application starts to scale and then are you do can you find workarounds for the time-based coupling and do you want to use a different database for a specific microservice so my uh my goal here is to spark a discussion and actually hear what you have to say about this like how do you do it currently and i want to spark a discussion about the fact that databases have made huge leaps in the last 10 years and do we still think the same way about microservices or is that changing are you already changing your patterns the idea here is as well that it's time for databases to adapt to your servers or microservices needs instead of the other way around and when a database actually scales and is consistent the trade-offs change significantly therefore it often depends on data requirements so uh and finally choices like whether you need database sharing or whether you're going to adopt a non-sharing principle and start a database per service could actually be different per group of services it's not because you decide that for one particular set of services going to have a database per service that you have to do it over all the all the services it might actually depend on what service needs what these are the actual ideas that i wanted to convey today and have a discussion about so i hope there are some questions in the q a after this if you're interested in how fauna could help you implement microservices you can actually try fauna for free we have a pretty large free tier so in that case go to fauna.com and if you have questions join our community at community.wanna.com you