Scale By The Bay 2021 : Eynav Mass, Prepare Your System To Scale OR Why Auto Scaling Is Not Enough
[Music] so that's a great question thank you for that actually we wanted our customer to feel as light as an antelope and because uh if you are a marketeer or a website owner data analytics can be so far away from you and so a heavy thing and we want to make that easy to consume so you feel light with your data as the antelope and you will be able to take decisions so quickly and fast as this amazing animal i hope it answers your question beautifully enough go ahead with your presentation thank you thanks everyone for joining it's so great to be here and i'm super excited to share um my thoughts about how to prepare your system to scale so [Music] i may now i'm leading the amazing group of r d to ruby for two years now but my real job is being a mother for three these three amazing uh kids at the picture and today i'm gonna talk about scale but you know sometimes it feels like everyone talk about scale right we hear it from all around we read about it or post about it lecture about it scale scale scale around i'm here to talk on a different angle that is not mostly discussed and that's the part of the system that cannot be easily auto scaled why i'm telling you that i'm telling you that based on the experiences we had at uribe so let's take a moment to explain what is aribi and why we are having scale and handling such amounts of data orabi is about codeless data we track events on our customers website we actually capture everything without any sampling from these events at our backend we create large data layers that describe everything that happens on the website in a structured way we modelize that in a way that we can create insight on top of it once we have the insights we can create visualization visualization that our customers can easily get from our application and dashboard and eventually our customers can export this data and leverage that in different and other third-party integration if they want to base their campaigns on that or create audiences based on this useful data you can read more about arabia at our website but now let's go back to my main focus and that skill so as oribi is about so creating data layer we have a large amount of data at the past year we started with 100 millions of incoming events a day and then in one month we just jumped to 300 event 300 million events a day and actually there was a one hour that we just had an increasing scale peak of 10 x in one hour and that was a huge thing for a startup for a system that is a bit new at the the neighborhood of big data and scale and although we have auto scale micro services based architecture and we had like a great optimized infrastructure it was still a struggle why because we had different angles and different technologies to help to handle we had the aws cloud provider services we had a kubernetes cluster to handle and manage uh through this scale we had our micro services apache kafka elasticsearch apache druids parts so many data solutions that we need to handle and manage in such a huge scale peak that was not easy but we did some great stuff when we designed ruby that helped us to manage that scale and to be at the place our day when i can share that with you and and tell about that in a successful way of the story so here you can see skeptic peak at one hour as i mentioned we had 10x of incoming traffic to the system but what's more important that in overall due to the success of the business we also have incremental scale each month if you add the incremental scale that we had monthly along with the huge scale peak that we um experience in one hour and combine it you get a lot of stress and high increasing scale on the system from different directions how to prepare and how we prepare again where a small startup we don't have a lot of engineers we're new in this neighborhood so first get ready for tomorrow today my first advice to you if you are in a place even if you are a start company or if you're a big company try today to examine your system and i'm gonna advise you with three pillars that i think that if you would um examine your system according to them you can really manage your system throughout the scale um i wanna illustrate that okay if you have just 100 and you need to take decision to prepare your system to design your system and you know that eventually there will be scale because you hope that the business will succeed you have a lot of solution that will cost you ten dollars and two solutions that you will cost you fifty dollars what will you do will you pick many solutions that cost ten dollars because you only have hundred dollars or will you go to the solution of the fifty dollars and you will have just two solutions that are more expensive i'm telling you that because this is the decisions that you take especially when you grow out your business and my advice to you don't hurry up to take the smaller 10 solution and maintain the 50 dollar solution on your table i know they're more expensive but it might pay off i'm gonna examine that and explain that more in a bit three pillars that can help you plan your system out for scale elasticity as a mindset batch methodology and isolation verification let's see each one of them elasticity my main focus here is that auto scaling is not enough i see many engineers saying that their system is so great because it can all the scale and as we go to data solutions it's not so easy to auto scale most of the time if not always you will need to do manual actions to scale out your system it won't be automatically and that places that exactly the spots that will harm and hurt while scaling up here is an example on oribi we're doing a heavily usage with apache kafka because we need all our system to be very elastic and we manage our messaging and traffic incoming traffic in a synchronously way so what we do we have a large clusters of kafka with a lot of brokers and we had a playbook saying when the scale will go up we will just add another broker to our class to our kafka cluster and just then we can scale out the cluster easily and here we solve the scale problem we won but what happened in real life in real life when the scale got so increasingly high the brokers were very loaded their cpu almost reached 100 percentage on that stage the cluster was loaded and busy when we added a broker we needed to do an action to balance the data to the next to the additional broker that we just added such action takes cpu as well so it's loading the cluster behind its current load if we would do a balance in such stage the total cluster will just collapse so we couldn't do anything we couldn't scale out because we couldn't rebalance the cluster with the new brokers that we just added and that's an issue what we should have done we should plan the playbooks better and start scaling out we before the cluster reaches the point that we can't really do any other actions like rebalancing the cluster so first i advise to you locate non-elastic components at your system usually these are databases data stores message queue or even network components like load balancer or other badly planned services at your architecture such services are really hard to auto scale so you should be aware of them and plan their elasticity accordingly consider the the ability of each component to scale not only according to the compute resources cpu but only consider memory storage i o concurrency and the network when you plan the metric that you will help you to take decisions of when to scale out or when to scale back in the next pillar is batch processing when i was a young engineer i thought that batch processing is like a luxury tool uh for us in the engineering team because it seems to be such an expensive solution but it's not that's the tool that can really help you to grow and to manage scale when the scale increase let's see that in a life example here is a data solution let's assume it's a data store and many services that ingest right into the data store in delta think that there is now a load and scale and all the services write a lot of data to the data store all at once the thing that the data store have a limit of i o mostly or all of them will break in some point so you will get services fail to write to the data store or badly the data store will get to some point when it's not available or even collapsing crashing what you should do in that case you can add a batch component so the services will not write directly to the um to the data solution but there will be a middle uh proxy component and you can handle when the ingestion is happening when the processing is happening and when to actually write to the data store and it won't be done in a full real-time process so um when load increasing on your system streaming is hard to control and it's more complicated to limit the cost and complication of changing your system architecture to work with batch processes can pay off especially on that spots but if you think that you will be handling scaling some point even in four years from now think of batch processing ahead of time this is something that should be really considered and designed because if you would like to add it once you have some scale drama or issue in the system that will be the last magic trick to pull from your heart because it's not something it's not a straightforward thing to do you need to take action to make it happen the third pillar is isolation or let's say how to avoid starvation here is another example at our ruby our customers have dashboards these dashboards is a random react application state-of-the-art application that what it does eventually writes talks with a back-end service that queries our data store data solution so when there is a customer that might have a lot of data at all um when he loads or her she loads the data throughout the dashboard it will create a large query on top of our data solution if you add the scale to that a scale of right scale of reads on top of that same data solution eventually it can create a starvation another customer so other customers might open their application and get timeouts or wouldn't actually load or eb application because of that so we avoided that by locating places where where our system components are not isolated and defending them accordingly the parts that are of risk observation will be the first to get blocked by those users that have a lot of amount of amount of data and this is something that you need to consider if you are in the early stages of business a starvation situation might sound like a fairway possibility to you something that is not really close to you however you should challenge your mind to imagine the effects of scale on your system leaving the starvation possibility on the table and even if you can't really get prepared to that at this point at least have a plan or at least design how to fix it later on so we talk about elasticity as a mindset batch methodology and isolation verification three pillars that can really help you prepare your system into skin i want to talk on another amazing thing that is super super important it's not one of the pillars because it's really related to each one of them observability when it comes to observability and data solution services application you need to be able to understand what's going on with them to detect the scale on time especially when you are a growing startup and you're still small you probably don't have two resources to invest in creating a large batch processing mechanism and auto scaling methodologies have full elasticity in your system right you don't have the resources maybe you don't have the knowledge or maybe you don't have the time so in that case at least invest in observability when i talk on to when i when i mention observability i refer to many metrics uh it's not only the cpu and memory that most of the time we check and examine there are many many different categories metrics that you should use to explore your system ability and your system health cpu and memory of course io network storage concurrency and even more depends on your system for example if you use message queues rabbit mq or kafka you probably would like to measure the lag if you are using a data solution with a hoteer you would probably want to measure the average query time such government can do and higher observation of the system health but i talked about systems so i want to give you an example of observability on top of system and not only on specific data solution because when we do an ad hoc monitoring for example on your elastic search cluster on your services on your kubernetes this is something that is really scope we need to also monitoring the system so we would be able to understand how the components behave with each other once the scales raising up here is an example you see here an example from ruby monitoring system we're using rafana and prometheus as i mentioned we have a lot of incoming data and some interesting metric that we are using in our ingestion chain that digests a lot of data in real time it's the end-to-end processing time this is super important because eventually if i measure every different component separately i might see i might see that there is some load on the elasticsearch cluster there is some spike on the apache drip cluster that we're using there's some memory or cpu consumption that is not so uh regular on apache kafka cluster but i won't be able to connect all the dots here on this metric we are measuring end-to-end processing that help us to connect all the dots because if there is some scale on the system if there is a delay on the system and it's affecting the system health this number will increase rapidly and that can help us take conclusion fast to react fast and to prepare fast especially the parts that cannot be auto scale and required some manual action that was it uh we talked about the three pillars elasticity batch processing isolation and on top of all of that observability because we need all the time to understand what's going on with the system even if we don't have still all the best practices and methodologies we handled everything that i mentioned into arabia and we had the great experience we improved the technology from scale to scale and i'm reaching you to scale up your system because eventually it means that your business is succeeding you have more customers and more traffic and that's a good sign i will be really happy to discuss with you in the chat room and you feel free to contact me in the linking i hope you enjoy that thank you very much uh thank you enough we still have time about nine minutes uh i do not see any questions on this card so people listening and do awesome questions and while we wait for questions uh i have one for you you know so you use a lot of technologies that in the end are using jvm so this uh jvm startup timer concern while you were doing your scaling can you repeat that again about the jvm please so we are using a lot of technologies which assume services and communities using machine so like you know you mentioned uh spark and a few other things in there so when you're scaling up your instances are jvm instances have to pay the cost of jvm startup is that being an issue for you for doing your scaling so first i think that's a good question because uh you're also taking in account the actions that are being done with scaling out right we're adding another service we're adding another uh kubernetes another food or another instance and this entrance probably needs to get some docker running jbm running this is the exact thing that i'm talking about taking all the actions that requires when adding another instance while filling out until it's really ready to accept new requests to the eject new data we take this in advance so our scaling decisions are not being done when we reach the 95 percentage it means that the system is kept in utilization that gives us enough space to act on time um i will give an example i mentioned the kafka so it's same for java services if you would decide uh when the current services reach a concurrency of 95 percentage to add another instance to add another service instance it's probably too late because as you mentioned there's the warm-up of the jvm and it might take time until the machine is ready so uh it won't be ready on time and request might fail and you might even lose data so my advice on that is to put the metrics in a in a threshold that will give you enough time to act it's probably 95 it's bad 100 percentage is worse so you probably should be kept in a round of 80 percentage 85 some companies even do 60 uh because they know that there is a heavy lift of the warm-up of the system and they want to be able to behave and act on time excellent yeah that's definitely something to remember we cannot scale at the edges you have to scale early and like enough said quickly so i do not see any more questions uh when you were scaling up and learning all these lessons uh did you get increased cost for a little while uh yeah we did get increased cost for example some of our machines were running on spot instances so spot instances are aws instances that are temporary you don't have a guarantee of the time that they will be up running live hence uh there might get down after a while after a few hours after a few minutes you can't really know and on some parts of the system such behavior created instability because if you take for example elastic search and running that on instances esports that are not all the time live and even though you have a replica factor that ensures that the data will be replicated on more than one server even though once the spot uh was down uh one server of the elastic went down and the whole cluster took time to get rid and we had a bad instability in the system so one action that we do is to replace uh spot instances for the elastic cluster and for more data solutions that must be stable and can't handle these downtimes uh and that cost us because this kind of change for moving from that type of instances to uh instances that are on demand but are all the time there but our most costly uh affect our costs um that that one thing and the second thing um we we did add more resources to the system but the good thing is that we have the ability to scale in not only to scale out so once the scale was done and we the system was mostly stable we could really quite quickly optimize back where needed and to increase um resources were not where it wasn't really needed to be with high resources or with really large instances i hope that answers your question [Music] you