Scale By The Bay 2021 : Natan Silnitsky, Battle-tested event-driven patterns for your microservices
so welcome everyone my name is nathan solonicki and i'm back in infrastructure team lead at wix.com and wix has a lot of scale huge scale and we have created our own event-driven architecture for 2200 microservices in production and this talk is about the key event-driven patterns that i have implemented or have witnessed implementations of at wix that have enabled us to create a robust distributed system which can easily handle increasing traffic and also storage needs so a few words about wix we have more than 200 million registered users from 190 countries more than 100 million websites have been built on weeks more than five percent of all internet websites more than 200 microservices and 2.5 billion kafka messages produced every single day so we have a lot of different event driven use cases that we can learn from and find the best patterns so in this short talk we will talk about four different evangelion patterns the first first one is consume and project how you can split up monolithic design and use events in order to be more flexible in deployment with microservices eventually from end to end is the second pattern how we can deploy kafka together with websockets to get to a pure event-driven architecture all the way from the browser to the end of your back-end flow zero latency key value stores is an interesting case of using uh having storage that you can easily query with because it's in memory that is populated from the kafka topic and is also continuously updated from it so really interesting use case and also events in transactions so how can we guarantee that processing happens exactly once which is very critical for some some parts of our business like with payment flows and stuff like that how we guarantee that when we process our events it only process them exactly once okay so we used to have the monolith um as our building block and we decided that we want to deploy these services separately and also scale them separately the different areas of our monolith so we split them up to a distributed micro services environment now if we have launch big share database among these services then we probably have mutable state that is shared which is is bad because there could be a lot of simultaneous work that can perhaps make our cause data corruption and there could be a lot of contention for the same database instances and also it means that if you use the same database for all our microservices it means that we won't have a flexibility of choice of the using the best database that our microservices requires so uh with microservices we've encapsulated the data behind our microservices that's great we have flexibility you want time series database go for it you want a graph database why not and and also the schema evolves separately because and it doesn't hurt uh to change the scale because it's hidden behind your api the microservice so if you have a relational table you can change that to your liking and no one and you don't need to talk to any other team about it now it look it's looking nicely but what about communications and happening between these microservices so we have a classic ecommerce flow with a checkout service and it requires to communicate with payment service user service inventory service in order to complete a checkout so with the request reply model you will have these http or rpc calls coming from the checkout service as a client to the payment service or user service as a server now this request reply model can be quite slow because now our databases are hidden behind the microservices api and if you need data from all these different services then you need to gather them one by one and by and then aggregate all the information so that can be quite slow not only that we can have sometimes network issues between to a specific service or maybe it's currently unavailable for some reason has a problem so that's a big issue so what will we do with our incoming requests that we need to process and we could have a cascading effect here where one service calls another service called another service called another service and the most downstream service has an issue so now the error will propagate all the way back and we have a lot of timeouts in our system and what if there's a momentary error and you want to overcome it so you'll need to create your own retry mechanism in your service to make sure that you end up successfully getting the data that you require so a lot of issues in terms of networking here with the request reply model now let's see if we can gain something by actually introducing a message broker in between the services so here we have a message producer and message consumer are like before we have a client and a server and by only just introducing this broker we can already really increase the reliability because kafka has because sorry a message broker usually will have replication and will have a cluster of many brokers that that replicate the information between them so once the message is produced we can be safely sure that it will then eventually be consumed now interestingly when you work with event driven model the flow control is no longer controlled by the sender here is actually controlled by the consumer so actually when you do request reply then the server is the one that is in charge uh it needs to gather up the response and get get it back to you and if it's unavailable well what can you do now here actually the server that we talked about in a client server is acting here as a producer so it's producing events and then forgets about them it doesn't care who consumes these messages or events its role is finished now it's the responsibility of the message consumer or the client in the client server architecture to actually consume the message and be in charge of what to do with it so there's no interaction directly there's decoupling which is always a good practice for your architecture and the consumer will be in charge of actually deciding when to consume the messages when to process them and what to do about them so specifically with kafka as a message broker we have a very sophisticated design which is based on a simple mechanism of distributed log files so each topic is just made up of partitions which are commit logs log files and a message is produced to one of these these partitions and while processing will be done sequentially on each partition you can increase your parallelism and work quite easily by just adding more partitions so your scale can grow linearly now the producer will just append the new event or message at the end of each partition so it already gives you a sense how how easy it is and fast to produce messages because you just append them at the end of the file and these messages can be kept indefinitely unlike traditional message queues where messages will get removed when the consumer actually consumes them and processes them and this immutable structure is great because you can have as many consumers you want nomad and you can start them up whenever you want and consume these messages separately so each consumer and kafka has their own pointer their own index of what is the current messages that they are processing for each partition and once they are done with the message they then commit it but it's not removed from some queue it is just kept in a journal that this is the last committed message for this specific consumer group and then when it restarts uh because there's a new version for this service or that has the consumer or something like that it knows exactly where to start from and this is a very powerful concept this immutability it means better performance better resiliency and much less data corruption unlike traditional message queues so like i said you restart and then or there's some network issue once you you gain your footing back and everything is okay in terms of technical technical stuff you just go back to process from where you left off it's that simple so you eventually consume the messages and process them okay so let's go back to our e-commerce example and here we have our checkout service that requires uh to have a lot of data found in other services in order to complete the checkup now let's look at how it will look with event driven model so with the ventricle model we'll actually have specific topics that each of the services is in charge of so the inventory topic is in charge by inventory service and it will produce updates about what's going on with currently with the inventory and it may listen to other topics in order to update the inventory but you as a consumer can know what's going on with the inventory by just listening to this topic and you can create your own caches in your service in order to know what's going on with specific elements that you are interested in now so checkout service listens to inventory topic car topic use this topic and then it can know all get all this information without these services even knowing that there is a checkout service it's not interesting to them and then the checkout service can actually actually be in charge of producing new events to orders because we want to have a new order in place now that we want to complete checkout so the payment service will listen to these order requests and then it will process uh the request so you can see we have a much more decoupled system here with services not knowing about each other and actually saving the data that is interesting to them and close to their heart to their geography now this is a very high level overview and i hope that the patterns that we're going to discuss now will help you understand more and get you more into this concept of the event driven architecture so all these patterns are used daily by wix microservices and i wrote a blog with a few more patterns that i will share with you the link at the end of the talk so i these are battle tested patterns that you can be safe to to use so we're talking about consume and project how we can create materialize the views for our database and from using kafka topics in order to make sure that we have the data optimized for for possible querying by different consumers and the users of our data and we can also split up the monolith by doing that and we'll see what it means eventually from end to end we're talking about the pattern how we can have streaming of events all the way from the browser to all of our microservices ecosystem by using kafka and web sockets together z latency key value store we're going to talk about having storage that is continuously getting updated and easily and very fast way of fetching this information in memory all using kafka and how we can process these kafka events exactly once for inch critically important business flows okay so we're talking about consume and project pattern here we are seeing uh one of our more important basic services that handle a lot of basic infrastructure for for our different microservices it's called the metasite service and it is in charge of a lot of flows for instance which sites are installed which applications are installed for each site what are what is the site version who is the site owner so a lot of metadata that's related to wix websites and this data is very important for a lot of our services other verticals like quick stores wix bookings and wix restaurants now metasite service used to be a lot more monolithic and it had like one big database with all of the large made up site objects with all the information put in it so it it got to more than one million uh requests per minute on this database for reason right and it became too overloaded so what did the native side team decide to do they decided to take one of the most popular queries about which apps are installed in the site this installed apps context for the site and split up all this traffic from the message service so create start splitting off microservices the first one is for this specific context so how did they achieve splitting it in a safe way in a performant way we started off with producing any updates happening on the meta side database to kafka so we just have site updates events and of course all the historical information also is kafka events then they created a completely brand new service called a reverse lookup writer and this one just consumed these kafka event updates for these meta sites and it's only interested in updates regarding site installed applications so it though we just filtered for the messages that it is interested in and only took the parts of the message that it's interested in and we wrote them to a completely brand new database table they installed apps context so they consumed the event and projected it to the dimension that they care about not the entire site just the apps installed on it now we want our users our wix restaurant service etc to actually be able to query and get this information so for that they created yet another microservice called the reverse lookup reader and this service accepts rpc calls and just reads from this database and answers questions queries okay so we had we created two new microservices why not just the one so we decided to the the meta side team decided to split the read and write service in order to gain flexibility so you can just take your database your new database table and replicate it for read-only queries so we have new read-only database instances and those can be scaled up and deployed all around the world as many as are required from the re-traffic load but you really don't need so much load for the right flow so you just have the original master database right flow and that is handled by the writer service but the reader servers can be deployed around the world and just read from the read only replicas of the data so it's much more flexible deployment scenario and it can reduce costs so you don't need to have the right side scale up as much as the read sign and what this database table has created is a materialized view so it is actually optimized for exactly what the query is looking for so the queries are much faster now and because we just produce the method so it just produces these events and doesn't care anymore about serving requests then there's a massive load reduction on the original meta site okay so medicine was much more happy and needed to do a lot less production incident handling after this split and you can also use it maybe in your use case or maybe you have a monolithic use case so let's look at second pattern which is event driven from end to end so imagine you have a long running asynchronous business process and for instance we're talking here about wix wanting to offer its users the ability to import contacts into the wix platform to manage all the site visitors like importing contacts from uh gmail or stuff like that maybe sending them announcements what's going on in the in the site etc now this process can sometime takes a long time depending on the third-party api and we want to keep the user updated on this context importing flow so we can we can do this all with request reply style and keep like status in the database and have the browser keep requesting to get updates but how can we do this completely in event driven style without any polling so let's see so we have two services involved we have the context job service that actually creates jobs of imports like splits them to chunks these jobs and then we have the dedicated dedicated import service so we have an import service for gmail import service for microsoft and dedicated and knows what it needs to do so how do we do this event driven so we can have the browser subscribe to get notifications from the http 2 websockets so we have a channel of the websocket the channel id of the websocket will be part of the request to start the import process so the jobs service gets a request to import csvs and or from gmail et cetera and it also includes a channel id so it will split up this request to specific chunks of data and it will produce these commands to kafka including the channel id then the actual importer services can consume these events and start actually the importing process and they also get the channel id where they need to return notifications to so once they are completed they can send an event to the websocket service hey please tell the browser if notify the browser that i have finished importing and then and here this is the channel id that you can send a notification on and then the websocket service will return the notification to the exactly correct browser now what's interesting about using kafka in this case is that it can really help you if you have cases that require cross dc communication for instance maybe there is some source of contacts that is only found in asia and you want that the importing done from that source to be done only in your asia data center so that's not a problem you can make sure that you produce requests to import to kafka and have it replicated to the correct data center and then these consumer will know that it needs to handle it because it's relevant to this asian source of contacts and it will handle it so everything is done completely seamlessly here so what have we learned we learned that we don't need to do any polling or create complex status database tables okay getting polled and maybe we'll need to do optimistic clocking on that and all these annoying concurrency issues that we have at databases um we simply notify the user on any status change via websockets and it's really scalable a really scalable solution you can spin up as many importer services that you require and then create as many jobs as you require and they will just each one will take the the correct job and work on it completely decoupled completely unaware of each other and then we can easily set up replication between our data centers between different clusters and easily set up the geographic specific location for context importing if needed so it's really easy to add cross dc communication great so we have our modern event driven style all the way from the browser to all of our microservices that's great and third pattern is the zero latency key value store so we're talking about a special kind of storage so you can think about kafka topics not only as topics of events that you want to consume and then maybe after a week old old-time events are deleted we can also have kafka topics have the partitions be like storage so they can be compacted and only keep the latest value for each key so there are key value stores you can think about using kafka topics as key value stores and you just set them for compact mode and then if you have these kafka topics use a storage you can actually load them into memory really really simply and and then you will have uh zero latency key value story and memory so we at weeks have implemented our own key value store reader implementation that simply has an in-memory map and a kafka consumer and this consumer reads this compact topic and it loads everything in into memory on startup and it sees that key one has a new value with v2 so it updates the in-memory map and when everything is finished you will have an up-to-date in-memory map that you can start using in your service in memory so it's really fast access now let's look at a few business examples using these key value store so for instance our business manager service that helps wix users with managing their business has a key value store for all the countries that wix operates in so you can like choose your country in the drop down or something like that so now all it has to do is create a compact topic and start loading the information into memory and booking service is in charge of wix users that offer schedules like yoga instructors and stuff like that it's like bookable lessons and stuff like that and it's very important when scheduling stuff that you know the correct time zone so it has here a dynamic configurable key value store of the available time zones for each country and we other than readers we also have at abstracted key value store writers so the business manager can update any time that a new country where works operates in it can update this key value store what happens behind the scene is basically it's just a kafka producer so we have a new country like south sudan so we just produce this key value to the country's compacted topic and then interestingly we can have seamless updates to other services that are interested in consuming these countries and have them as part of their in-memory key value store so bookings is also interested in the country's key value store so it has an in-memory kv store reader and has a consumer for the same compact topic and it gets an automatic update whenever it's updated including here with south sudan so now that bookings has accepted this new event that is a new country south sudan it can actually now automatically add the time zone needed for south sudan so it also has a key value to rider for time zones it it finds the correct time zone for south sudan puts it into produces this event to kafka and now its key value store reader for time zone is automatically up to date with the time zone for south sudan which is great and of course other services that are interested in time zones can also set up their own key value storages with their consumers and get their up-to-date time zones automatically as well so you see this like uh interesting event driven pattern that the producers of these events don't care who is going to consume them but these consumers can get automatic new events because they're listening to updates and so your data is can be spread around all of your services and be used seamlessly without needing each service to know about the other service completely decoupled so what have we gained we gained loading compacted topics to memory which reasons we have zero latency key value stores where we can put dynamic configuration and it can be accessible for any service that is interested in it compacted topic is still a topic so other services can consume it now an important note here is that this pattern is only useful when you have small data sets that can fit into memory because memory still costs money it's not infinite so you want to make sure that for large database sets you don't only use an in-memory solution but you actually stream this formation to a more persistent store at wix we actually use aws dynamodb because it offers infinite scale and we can put when we have very large datasets we put the information in dynamodb and then we can stream the events from the nbdb using dynamodb events but that's out of the scope for this talk and i had other talks that talk about how we use dynamodb together with kafka and our last pattern for this talk which is events in transactions how to make sure that you process events exactly once in an item potent fashion so item potency means that no matter how much how many times you try and do something with the same information you won't change the state of your data so it's kind of a deduplication something it's item potent it it it doesn't change no matter how many times you run it so how can we make sure that we are able to work even though we may have messages that potentially get produced more than once so usually that is handled by the application level but let's see how kafka itself has handled that for us in order for us not to need to do this item potency duplication work in most of our flow so we can go back to our classic e-commerce flow and we have a payment service and it wants to announce that payment has completed so it produces it to kafka now we have our famous checkout service it consumes the fact that payment has completed and now it wants to announce to the world that the order of the checkout has completed and it is enumerate enumerating all the items that are part of this order now we have a lot of other ecom services that are interested in this event we have delivery service that now that order has completed it needs to deliver the items that are found in the order inventory is definitely very interested in consuming this event because it needs to reduce inventory per each item that has been paid for and invoices service is also interested because it needs to send all out invoices for each of these orders that are completed now we really don't want to have delivery service trying starting to deliver items twice right or inventory being reduced twice or setting two invoices or three invoices because we unfortunately got the messages more than once so how do we solve it one way to solve this is to like i said do the due duplication by ourselves so we have these services need to set up each one need to set up their own completed orders table and if it gets like the order id in the event it keeps it in the database and then it needs to make sure that it will write down that this order is already arrived and don't worry how many times it was produced by payments or by or by the checkout service you you'll need to first access the data and see uh whether it was already successfully completed or not and um each of these services will need to to handle deduplication by themselves now let's see how kafka can help here the kafka producer you can enable item potence by just reading a flag and then it will make sure to produce messages exactly once how will it do that it attaches the offset of each message to the message so let's see how either important producer works it produces a message including the offset so then kafka broker says okay i got a message this is offset 0. great now the idep of the producer will increment uh the offset for the next message and it will produce and say okay great we have one we are getting the message and it's offset one now if for some reason the producer will send the message twice it will sell away with the same offset so the broker said aha i see that offset one i already have it so i'm just ignoring this message i already have it i don't want to get any duplicates now for the rest of the streaming here of the events processing including the consumer and the producer in the middle and then the downstream consumers you can actually set up a kafka transaction i wanted to get into too much detail because it's it's too broad a theme but you just set up a kafka transaction and then uh you can consume it and the consumers the downstream consumers will only consume messages that have the transaction has been completed so even if there was some error and messages were perhaps getting twice the transaction will be aborted and and the transaction will start again so all the messages before that will be disregarded and only when transaction is completed then the consuming will start meaning that we get messages to our downstream topic exactly once so what what is useful in this transaction so first of all these transactions are quite complex there's more boilerplate for producing and consuming in this regard and you will need to do performance tuning you have a trade-off between the overhead of this transaction and uh with the transaction size so the smaller the transaction the bigger the overhead will be for maintaining this transaction but the less of the latency will be for the downstream consumers actually reading this transaction now the bigger the the amount of messages in the transaction then smaller overhead of maintaining these transactions but the higher latency and of course the downstream consumers the final consumer has some side effect outside of kafka so kafka can't really handle that exactly once because it's the responsibility of the processor at the end but you can actually do the duplication by using the kafka record offset as your version so it really simplifies the duplication at the end now i wouldn't recommend turning on transactions by default but only use them in very specific cases like payment processing and before moving forward just wanted to let you know time is uh running out so we can maybe do the rest and speak yeah yeah this is one of the final slides so if you want to understand more about exactly one semantics in kafka i have a dedicated talk for that and i put all the links for you don't worry about it so just to summarize wix developers have employed all these event-driven patterns to make their microservices more decoupled more resilient and more scalable in order to have a mature distributed system that is deployed globally and is safe to operate and you can also check out our great high level sdk for kafka if you're using the jvm uh which calls greyhound it's had a lot of cool features uh for performance improvements for resilience improvements it's really great i really i hope that you check it out and i would like to thank you very very much and here i have extra resources for you on the slides so about event driven architecture my blog posts a great set of um of blog posts on how to think about your microservices when you do them in avengers style and the talk i gave on exactly once semantics and i put all of the slides with all of the links i slide share so you only need to look me up on slideshare not on sunsc and that's my name there and you can follow me on medium and twitter to get all kinds of updates on everything we do on infrastructure and weeks around event driven architecture around kafka around caching everything to do with software engineering i give updates on twitter and also linkedin and of course you can visit my website natanzil.com to get information on all previous conference talks i gave blog posts and and that's it so uh thank you so much and partis do we have um time for q a during the session or we will need to move over to special chat you