Scale By The Bay 2020: Natan Silnitsky, 8 Lessons Learned from using Kafka with 1000 microservices
[Music] so uh welcome everyone my name is nathan slanitsky and i'm a backend infrastructure developer at wix.com and this talk is about the key lessons that we have learned over the years as we scaled up our kafka based distributed system to more than 1 000 scala microservices and you learn about the best practices tools and sdks that we have created in order to achieve this so a few numbers about wix we have 190 million registered users more or less from 190 countries 5 of all internet websites around wix and we handle more than 500 billion http requests per day and serves six petabytes of static content now our kafka messaging has a bit lower scale where with 1.5 billion messages per day but this scale is is quite large for our micro inter microservice communication and we also have other scaling concerns so we have all in all 1500 microservices 1 000 of them scholar based and 900 developers with all kinds of use cases that we as an infrastructure team at wix need to support so what do you do when the traffic metadata and amount of developers and use cases keeps growing so the first lesson for this talk is about the need to have a common infrastructure with common features for this massive amount of developers and use cases so all of you are probably familiar with the kafka producer and kafka consumer that are provided in order to communicate with the kafka broker so we decided to wrap that for our developers at wix and we did a wrapper called greyhound which basically has a simpler api than kafka sdk and we added all kinds of additional features that help our developers achieve what they need to do and it also serves greyhound also serves an abstraction layer so it will be really easy for doing changes like upgrading kafka version or fixing a bug or introducing something new and easily spread it to all of our services in production very easily and we recently rewrote greyhound on top of the zeo concurrency library which was great fun and we also have a java api on top of it so scala future based api zero-based api and java api and let's see a few of the features that graham offers our developers so first thing is if you look at how you set up a consumer it's java based so you start off and create a consumer you have you need to specify the broker locations and you have the key type and value type of each record so you need to specify the these serializers for them require a lot of other boilerplate and when you actually want to consume messages then you call the poll method from the consumer and you get back records and then you probably do some processing for each of these records and you call poll and process and commit method again in a recursive style in scala probably and if you don't want to lose any messages then you'll probably want to commit the message after explicitly after you process it because otherwise if you go with the default kafka consumer of auto commit you may end up losing messages in in certain rare cases but that could happen so quite a lot of boilerplate here and if you look at the how we do it with our greyhound api so the user needs to specify a record handler which basically just needs to get a lambda with processing of each record and also specify the key in value types and they will be automatically provided because usually you just want some json or in our case json or protobuf realization and we also and then you just provide the handler to the consumer builder with the topic and group and that's it and you don't need to explicitly commit the records or something like that greyhound handles it for you you don't need to specify broker location that will be injected automatically through the configuration file in production okay so a very important feature that greyhound adds that kafka sdk lacks is parallel consumption so if we go back to the vanilla consumer usage then we do the polling of the records but actually we can't do any multi-threaded access to the consumer we can't just if you want to have more parallel work do more polling and processing like that um because the consumer is not thread safe so with greyhound what greyhound does when it pulls kafka it actually takes all the messages and then assign each of them to its own queue all the messages from the same partition will be assigned to the same queue and processed in parallel from that queue using zeofibers so fibers are you can think of them as green light lightweight threads or green threads where it's really cheap to do contact switching between them and they take up a lot less memory resources than jvm threads so it means that we have very efficient parallel consumption out of the box for our users and we also have greyhound handling errors so we have a distributed system it's really important to make sure that even if an error occurs during processing eventually the microservice will able to recover and continue on so regard offers retries we have here a non-blocking flavor of a retry configuration for greyhound we specify i want to retry after one second or and then if that fails after 10 minutes of course they can also be also there is also a flavor exponential back off just provide a retro config to the builder and that's it you have it and how does it work behind the scenes so if the grand consumer fails to the handler fails to process the current message there's a built-in retry producer that will produce the message to the reach right topic and there's a retry topic created for each of the intervals that you will set in the retract configuration so retry 0 will process the message once the interval elapses and if there's a failure then it will then be produced to the second retro topic etc and the using uh working with retries using retry topics was inspired by an uber blog post that we read and we implemented on our site but there's an issue with retry topics that will they will cause your clusters to grow faster as you see we have a lot of retro topics potentially because users want to have many retry attempts and so this kind of exponentially increases the amount of topics and partitions that we have on our cluster so one way to mitigate it is to just specify by default one partition for this retry topic and then you will have less increase in the metadata that your kafka brokers need to handle and we saw that with retry topics of the non-blocking kind messages will be produced by retro topics and at the same time there will be also processed uh on the original topic so you get out of order execution but uh we do have a solution for that with greyhound where we offer a blocking policy where messages will be retried the same message will be retried again and again until successful processing but what happens here now is that you get a lag between the producer that keeps producing messages and the consumer that is stuck on the same one so for that we have a dashboard automatically created for each service where you have the ability as a user once you get an alert that the lag is increasing to go check out your the payload and decide if this payload is really not interesting and you can allow to skip it you just unblock it and move forward and hopefully the lag will go away but if you see that oh i think there's a bug in my code you can go fix that code and redeploy your service and then hopefully because you the the same message will be processed again because it was not committed yet and now uh hopefully the bug went away and also the lag will go away because the messages will get processed faster now okay and another very important graham feature for us at wix is context propagation so we have a lot of incoming http requests like you saw and our users have a lot of data in those http headers so consider the following signup use case where a visitor of a weak site wants to sign up and be a member of that site okay so what can we retrieve from the headers we have the what's the language of the user what's the geography is this user already a member or not or is this the website owner all this information can be gathered and handled by the site members service for the sign up request and if it wants to produce the message to kafka and then have other services consume and process this event then we want to keep the all these headers http headers and request context as part of the flow going around in all of my microservices so greyhound just serializes this information into the kafka record headers and then other services like contact service that manages owix users contacts can then consume this message and get all of this con request context information automatically and then use it maybe they want to store something that information in the database okay cool so that was a little bit of a greyhound we'll also return to it later in the talk and the third lesson i want to discuss the describe today is when you have such a big amount of developers and use cases to support you really want to have as much self-service as possible so the third lesson is have self-service tooling and documentation in place so we in the past we used to get a lot of questions on our slack channel like is the way to search for specific kafka message or i would like to increase the number of partitions to an existing topic and that is something that our team is in charge of developing infrastructure needed to support so we decided to have as much tools and documentation in place and in terms of tooling of course there's the kafka cli scripts that you can consume messages and configure topics but for our users are developers it's really quite quite difficult to get acquainted with each time and remember what are the flags that they need to uh need to have and they need to install these scripts on their local machine so having a web-based gui interface is probably better for our developers and there are open source solutions out there like the yahoo kafka manager and also commercial one like confluent control center and but we decided to create our own control plane because of our own special needs so here is a quick peek at our back office control plane for kafka so the first view here is to configure topics so you can view the current configuration for each of the data centers and then you can specify for instance increase the amount of partitions for some or all of the data centers so that's very handy for our developers and the very important view as well is to have a live stream where you can specify the topic data center and from each time you want to get messages streamed to the view and then you click play you get messages streamed and when you click on a message you can see first start which service produced the message which services consumed it at which times and you can also of course see the payload of the message with the nice formatting where you can copy parts of it if you need to and also of course the headers that are the record that can help you debug the production error maybe if there is one and another very important view for our developers is the key value stores view so with kafka you don't only have to use it as a messaging platform it can also be used as storage where there's the topics that are described as compacted topics where the last value for each key of the record is kept so you can think about it as a key value store so here we have a view where our developers can actually view the current key values and maybe understand what's going on in production at the moment okay so that was a quick peek at our back office for kafka and let's talk about documentation so there is questions like how do you investigate the slag right that's the big production issue with kafka growing lags and questions about how to use greyhound i want to add retries on errors so we created a very extensive github readme because developers are used to github for getting information we also add a lot of q a style data for stack our internal stack overflow site and also on our channel we have a slack bot that answers questions without us needing to be present so here's a look at our beginning of our greyhound readme and the q a style questions about how to investigate kafka consumer legs and we have our tuba here that tries to answer people's question from earlier questions on slack or from stack overflow and other resources and another very important topic about self-service is self-discovery of topics and message structure when you have so many microservices with so many topics you really need this to happen self-service you can't have some team always needing to get this information out by getting questions from other teams so for that we developed an a an api viewing site where we get all the messages or structure defined in protobuf for platformized services that are widely used and they are then published to the internal api site using swagger and then the developers can view all of the grpc requests and all the kafka events and everything revolves around the protobuf schema of course another very popular schema is the avro one and you can also use the schema registry from confluent and then get not only viewing of the current schemas but also the ability to manage versions of schemas so that's a very good benefit of using the schema registry but this is really helpful for our developer velocity okay the fourth lesson for this talk is that monitoring in a completely distributed and asynchronous event style system is less trivial than your standard request reply in a synchronous manner so for that we really need to have as many matrix available for our developers as possible so greyhound has messages built metrics built in that it makes exposed for the prometheus matrix server on the producer side consumer side and then we have automatic dashboards on grafana for each service it has a lot of interesting information for instance what's the currently longest running handler that the developer code provided that is currently getting stuck so we that can really help with investigations on the on the production error and also if the lag is building up and building up you get automatic alerts sent as a developer to the email and to slack so here we can see for example an alert that incoming to slack and developer acknowledge this in less than a minute so that's cool and they need to check why do they have a slow handler specifically stuck on some message cool so the fifth lesson for this talk is about proactive broker maintenance so remember at wix we have a lot of traffic and a lot of metadata a lot of topics and partitions and there's a apache kafka blog post on the optimal amount of partitions which is four thousands for each worker and and then each cluster they have up to 200 000 partitions so we took this into account and kept only adding brokers when needing and splitting cost clusters to smaller clusters in order to make sure that the broker is healthy and not getting stuck especially restarts can be quite long when you have so much metadata and delete unused topics when you don't need them anymore this allows us to avoid heart failures and keep the clusters healthy and we're migrating to confluent cloud and this really this quarter so this will make it this uh work of keep making sure the cluster is healthy and not on our hands but going to conflict and that they provide high availability so we don't need to worry about keep scaling the clusters and they're also doing a lot of work on improving their the maximum sizes that they handle so we're really excited about that okay the sixth lesson is avoid using kafka sdk directly in node.js so at the beginning we had only scholar services with the greyhound library but then as node.js services started cropping up we knew that we were going to need to have an equivalent of greyhound to the node.js so the js platform team created a gray node which they based on some javascript kafka library but it really wasn't an optimal solution it's a quite buggy library and they need to do keep catching up to what we did in greyhound so this really wasn't optimal so instead we created a sidecar and that a proxy for gray as a proxy service for greyhound where the node.js communicates with using grpc so this really helped us have new features in greyhound easily be provided to node.js services as well there is the downside here of having bigger memory footprint we have double the amount of containers now for each node.js pod and uh there also jvm takes up more memory than uh the node.js so we're trying to reduce it and considering moving from a sidecar container into a demon set solution okay the seventh lesson remember we have eight lessons is consume and project so this is talking about how to deal with legacy monolithic code so we have a meta site service at wix that is in charge of helping know a lot about what's going on on wix websites for instance it has information like what's the site version who's the site owner or what apps are currently installed on the site and that's really popular flow that's needed for all of our verticals for instance stores restaurants and other other services so the database uh became really clogged and loaded in high contention for reading right with more than one million requests per minute because it stored all of the meta site need in one object on database so the team decided that they're going to split some of the flow here specifically the installed apps contacts and they did that by first producing the entire meta site object context into kafka as an event as a site updated event which then allowed them to create a completely additional new service called the reverse lookup writer that consumes these events but is only interested in the app's context events so its filters unnecessary information and stores in the database only the app's context so it kind of projects the information to what it needs and creates a materialized view that is highly optimized for the query needs of our services and it also created an additional service that's read only that's called a reverse lookup reader where the rpc requests come to this service only which means that it's really easy to just scale up reading as if throughput and traffic we can have it all around the world and it can actually read from database read-only replication that is eventually consistent from from the master database table to the read-only replications all around the world and for the last lesson for this talk lesson number eight we know that kafka messaging is event driven but is it only relevant for service to service communication what about browser-to-server interactions can they be driven where but a user is waiting for a response right wrong we can have uh event driven work with the browser by using kafka and websockets so together they offer us a completely distributed and event-driven solution if we consider the following use case of a long-running asynchronous business process we can see here that the browser can request to do an import of contacts from google and microsoft to the wix platform for the week's user and they can do it like with request reply and then poll for the status of the importing of contacts are you finished yet are you finished yet are you finished yet and the service in charge will need to create a database with all the statuses and optimistic locking etc but instead of that we can use a websocket service in order to have this event driven so we have the context job service that creates jobs and once the browsers subscribe for notifications for the websocket service with a channel id for getting notifications back it will send import requests to this jobs creator service and this jobs creator service then creates these job events to kafka broker and then we can have a lot of services to actually consume these events and actually do the importing because of this decoupling we can have a lot of them but let's say we had just one service with a lot of instances that actually does the importing from google and once it completes the job it just sends the notification back publishes it to websocket service with the channel id of the browser that is came along for the ride on the kafka event and then the websocket service will just notify the browser on that channel so we have a seamless flow here that is completely distributed and decoupled and you don't need to add any more database entries and do polling so you save up resources and be a lot more efficient also so to summarize what do you do when the traffic and metadata and amount of developers and use cases keeps growing well at wix we created an entire ecosystem to support large-scale kafka related needs with tools with self-service libraries infrastructure everything that our developers need in order to have the best support they can and also our production environment to have the best debugging and investigation capabilities and robustness of course as well and i'm very happy to announce that greyhound is actually open source and publicly available on this github link and it offers both scala sdk and java sdk and the 0.1 version is out now so you can go and check it out and be and see for yourself so thank you very much and i'll put all this i've put all the slides on slideshare so you can go to slideshare and see the slides if you want to go back to some material and also you can follow me on medium and twitter in order to get updates on what we're up to all around event driven architecture kafka related topics and also zeo and functional programming as well and you can check out the previous conferences uh talks i gave on my website at nothingstill.com so thanks again you