Scale By The Bay 2019: Petr Zapletal, Change Data Capture in Distributed Systems
Recording: Scale By The Bay 2019: Petr Zapletal, Change Data Capture in Distributed Systems
[Music] my name is Peter man I work for for this name the last year I was here I was I said something like we are not a school from us but we just watched a new streaming service go go Disney Plus so I hope we at least have a school as they are now by the way the release was on on Tuesday so as you can imagine the timing of this conference was like not that great for me I'm here and now the topic of this talk change data capture and in distributed systems as you can imagine a system with belt is massively distributed we need to deliver content to millions of users around the world and we have to track data changes among various micro services all the time and this talk I would like to give you a quick introduction into CDC just in case you're not that familiar with this concept I have a feeling that people are not as focused on this as they should be and also I would like to share my experience with CDC and give you some recommendations systems can be categorized into the source of the truth systems usually call them like primary stores as an example we can have a relational database with data about users the place we go to when we need to update their name or addresses or doing squaring and things like that then we have derived or secondary stores if you will data in these stores are derived from primary stores often you saying are you transformations as an example from the user domain I just mentioned we can cache use addresses close to the entry point of the system and show users different content according to it and we would want to do that because I primary seek restore may not be able to handle do you expect a load for example so what CDC means it's a little bit more generic of course it covers a lot of use cases I will give you some some samples later the definition of CDC is that CDC is a design pattern that enables capturing changes to data and notifying actors so they can react accordingly when thinking about CDC and a lot of things to consider so let me give you some some ideas it's not gonna be an exhaustive list for sure but it should be helpful so let's start with reliability all dorrance today tell us essentially I refine a data and to write that data stores become inconsistent or is it this is something which is like absolutely unacceptable for you it very depends right then we have scalability do we care if a system is able to scale with increased load or or data size or we do believe it's not really needed and our performance do we need our changes to be propagated to subscribers in real time or is it okay to wait a little bit consistency do we care about ordering of events for example another thing to consider and our fourth Lawrence is the system to learn two failures we may or may not want to introduce some redundancy to make it more resilient actually we could find many more but I think this like a good good initial batch when implanting cdc's there are few strategies how to do so as you would suspect we have quite a few options so let's start with probably the most obvious one basically the records have a status attribute moves come on most common is it's a time stamp or a large number or something like that this approach is very simple and can be used almost everywhere as long as we can add this attribute but there are certain reliability concerns and as I will show you later this approach is often combined with some kind of polling to check whether if any changes have been committed to the requested record but I believe two different implementations are also possible for storage engines that supports database triggers like my sequel or Postgres we can implement store procedures ticket by various of operations those can be used to see muscly propagate changes to edit a data tables or even stores triggers are very reliable and fast but may drain database resources to the function essentially compares the current state of the data where the previous state of the data the good thing is it's like very easy to implement and it's like super generic on the other hand it requires a lot of resources to move the data around and you know it's gonna be performed real time a lot of data storage solutions maintain a transaction or a change log or an audit trail if you will know essentially the same thing this is used for recording and tracking changes from committed to the database in order to to recover the committed state you know for various reasons obeys CDC's take advantage of this by consuming these logs very reliable it has no impact on the database performance and it's very fast on the other hand looks are not standardized so it's often how to read these data and it may even like change between like two versions of one database which is like super annoying but it's becoming more approachable these days I wish you a real world example and in a minute so there was it was talking and now let's to do something more interesting how to implement CDC you sink polling or how can I call it like poor man's CDC so let's start with our server which is the service with access to primary data store in this example the store includes data about some rules but you know that's not really important for us at this point and as you may see the abuses play as is web server it's a pretty unknown open source framework level by live event we have part of you I have using this in production in this example we use play for implementing the HTTP endpoint which is head by clients to get the rules data you can also recognize that used to be for accessing the database Dubey is a functional JDBC layer it is really nice we use it all the time DB represents the story here and the function of federal rules just you know exactly what you expect you know something like select some fields from some some tables kinda the skip map snip I just transformed the original data to the format you know what expected by particular and this is how the client Paul's rules firstly defines service hostname you know this is the address which is hit every n cycles and here we use play WS client to do the get call to the defined endpoint and finally we update local state you saying the the receive response as you can see this is not ideal we have to hit the server regardless of if anything has changed or not also we are not able to update a state if the server is not available apart from that it may be quite resource consuming if a lot of clients so let's try to do something something similar this time has an audit trail slash rock scraping we still use sequel as our source but this time you want to send the rules to DynamoDB which as queried by the client I know and we do is sync of course as you can see we introduced I dedicated the table for it this table essentially represents the rules in the same way as the dining table we could make it work even without this you saying the original rule tables exclusively but it may be quite challenging to handle all the various events and to map them like correctly so we decided to simplify it and accept data duplication and its consequences to a limit its complexity so yeah do rule data are stored in this field in reality we we do have a few more columns here of course but the funny race this is not a relational model anymore it is what dynamo wants it to be D table is not special by any means we used to be again I just wanted to show you some some simple it's very straightforward as we are just updating a single table just make sure you do this in the same transaction you use for updating the real relational tables which you know I don't have here because the slides are not big enough for this as a messaging letter we use AWS VMs which is database migration service it's a service built exactly for this purpose it supports various sources and targets including Postgres I'm dynamo which is what we want here it is also a managed service so it's quite easy to set it up when you sing the MS you need to define values you know selectors and mappings and some other other things to make it work it's not difficult and it's a reason I play documented to be honest so it should be quite straightforward so at this point all the updates are flowing to to dynamo you know we just need to access it to do so we need to define and ADT as you can see it just corresponds to the table definition we have seen before and called the define get function which retrieves the rule data using car keys as you may see we use karma which is another great library I would like to recommend and that's pretty much it this solution scales of a better than the previous one it also decouples the services a little bit you know but it's a little bit more work and now I would like to give you some ideas about what is use cases usually based on what we use CDC for this again won't be any an exhaustive list by any means but I believe it could be you know it's powerful for you guys outside of the application we just have seen essentially changes from a baking data store like you know Postgres from my sequel are applied to a cache or a secondary data store you know like DynamoDB cassandra memcache Redis or something else this thing approach is usually preferred we have could find a few implementations of this in our case primary store are usually sequel CP databases wrapped as RDS and a secondary stores we use DynamoDB and Redis we have value search apps that we use for almost real-time indexing and use CDC pipeline to populate our search backends that that's usually elastic search these days the next use case I'm here is quite opposite instead of moving data caches or search indexes we can move our data big data processing systems like redshift or snowflake or Goodall s3 for further processing and analytics or just as longtime storage I think this is like super important these days as they are key asset for for many businesses you know it's great work on a greenfield project pretty much know everyone loves that but unfortunately that's not always the case with the seminal ethic apps often need to do more things per transaction similar to the example I mentioned before you may need to write to a secret a base and also update cache and things like that these are usually called dual rights and twelve rights are the known to be complex and fragile no CC helps fix these shortcomings by separating these rights the current query responsibility separation or or CQRS is an architectural pattern which uses one data model for writing and one or even more models for for reading this can maximize performance cable scalability and it's also like very very flexible but it also adds a lot of complexity CDC can make this more approachable so for example rights can be recorded to a database as usual and we can create streams based on these right events to dream the streams can be consumed by by the services reading the updates for their read-only views as I said this is definitely not an exhaustive list I'm sure you can find fame or applications actually I'm pretty sure we have way more of them just just Enders name but I'm going to give you some samples to inspire you and as you'd assume there's a bunch of frameworks trying to make our life easier I will start with a little promo it's name is PG to k4j and was developed and open sourced by us as an alternative to PG to Kinesis which we found not perform at enough conceptually it is quite simple there is a running Java process which reads Postgres s write a headlock transform the data and uses kpl to write to two kinases and if you interesting into this there is a github link in the references section Maxwell is quite similar it creates a daemon which reads my secret these are conceptually very similar to Postgres what I'd headlocks I just talked about and it rides to various streaming platforms including Kafka and Kenan say something like that Maxwell was still an open source by baizen desk then I have at this point well-known TMS and in the example I used EMS for pushing data from Postgres to two DynamoDB but it has very broader usages as a source you can also use my sequel MariaDB Microsoft sequel or even as three targets you can use Vario sequel stores there are also things like redshift aforementioned dynamodb elastic search MongoDB M some others this is a managed service which is good and bad and also it's very tightly coupled to AWS spinal tap is a general-purpose reliable CDC framework people by Airbnb it's built around Kafka so technically you can use that outside of AWS and I believe you can use Kansas to if needed it supports various data stores like my sequel or dynamodb Eddie's of around at CDC social I have the links in the end as well database is another open source EDC framework in this case stable by LinkedIn it is also a general-purpose solution it was developed and 2013 and as far as I know it's a replacement is already under development but it may be still viable for you and last but definitely not the least framework I want to mention here is the museum the museum is again an open source solution but it's backed by by Red Hat and or you know technically IBM these days the busy and implements unit even model and connectors for various databases it is built around Kafka and Kafka Khanna to take advantage of its great traits like durability reliability and photo and also this project is very active which is always a good sign as you can see so far a lot of options of course there are more available so like feel free to look around if you need professional support it may make sense to go for one of the commercial solutions which I haven't mentioned here but that's it's completely up to you you know always have in mind what makes the most sense for you in the micro services worked we live now CDC is surviving you subtle pattern which I believe everyone should have in mind I hope this helped you guys to build a little bit more structured view on on CDC and give you some ideas what the thing about that's all I have today thank you [Applause]