scale.bythebay.io: Martin Zapletal, Intelligent system optimizations
Recording: scale.bythebay.io: Martin Zapletal, Intelligent system optimizations
all right thank you hallo my name is Martin I work for cake solutions a van tech media company and today I'll talk about intelligent system optimizations there's a motivation we Channel which one of these two code snippets do you think performs better the one on the left the fold left on on the right the while loop yeah probably the while loop I don't actually know I don't think I've ever tried but it also throws an exception so I guess what I'm trying to say what I'm trying to say is optimizations are difficult and there's a lot hidden in the background that that's often difficult to identify but don't worry I won't talk about any of that throughout the rest of the presentation I'll talk about optimizations at a higher level optimizations using intelligence using using machine learning so most people when you talk about intelligence they assume we're talking about performance but we can optimize other things as well things like reliability uptime and cost is one thing we often want to optimize and there are different levels so we can optimize at the code level or logical physical level but we'll talk about the highest level using using intelligence I do have some tensor float slides in the talk but I only have 20 minutes so if you don't know tensorflow I apologize in advance I'll just go through it so quickly you won't even have time to be confused so machine learning these days is used in many areas it's used for stock market predictions object recognition or even self-driving cars many many interesting business areas very often when you go to a to a website or like an online store they track what do you do they track your clicks they track how long you spend on each on each page what do you search for they try to improve the your experience they try to optimize the experience for you as a user we're talking about something similar but using a distributed system so we're used observability data so data like logs metrics traces or anything the system for uses and will use the data as an input for a machine learning - a machine learning algorithm and tried to use that to analyze what the system is doing understand what it's doing and when we know what it's doing when we understand it we can try to define some actions and improve the system in certain direction whether that's cost or performance or something else we care about and all of these things mostly matter at scale so you know if you have a small retail shop and your your system goes down for a for an hour that's a major hit for your business but it will probably cost a few hundred thousand dollars few hundred or thousand dollars but if your system has dozens or hundreds or thousands of machines and it's a huge system and it goes down for a period of time it may cost you hundreds of millions of dollars and and the opposite is also true so if a google managed to improve the electricity usage of their data centers by fifteen percent so if your household is able to decrease their its electricity usage by fifteen percent you probably save a couple of dollars but for Google and the scale at the scale of their data centers they were able to save hundreds of millions of dollars a year just because of the scale so the scale amplifies the effect of any optimization we can do to our system and it often becomes about finding the balance not about we don't want to build a perfect system what we're aiming for is aligning the risk taken by a service with the risk the business is willing to bear so we're finding the perfect balance sometimes there's a known issue in the system that can happen very very rarely but it may not make sense to fix it because the impact is just not worth the effort that it would take to fix that problem but the important thing here is it has to be an explicit decision you can't just have a system full of holes and if it goes down just say you know whatever it was expected you have to make the decision explicitly you have to make know about those issues so you should still build all the good things in your system like reliability and delivery guarantees and and you know consistency and choose the correct consistency validated should all do that still but sometimes you may choose not to implement some things so machine learning is already used in many large-scale systems for various things so these are some of some of the applications that are already researched or even used in many systems such as known as node assignment cluster scheduling in the resource management these are common problems in large-scale datacenters serveral is instant instant sizing automated scaling or you know data mining the data to understand what your system is doing these are common use cases we'll talk about today about configuration which is a bit less common use case but it's nice to explain some of the some of the concepts this is a very simple Apache spark job it's a common machine learning pipeline just loads data from a file splits the data into two parts training and testing data set and then defines a simple machine learning model in this case a multi-layer perceptron and then uses the model it trains the model and then uses the the testing data to validate that models performance on a different data set it's a very very common machine learning pipeline the basic training of a machine learning model but there are two things to notice here firstly we define some attributes here so we define the number of layers we define some other parameters of the training procedure so how did we set those values and at the same time spark has a variety like a huge number of configuration values this is just a small subset this is around 20 configuration values but there's many more how do we set those what is the best combination so the first problem parameter and hyper parameter optimization we won't talk about there's a whole field of machine learning around learning to learn and meta learning but we'll talk about how to optimize these configuration values to improve the runtime of the SPARC job these are random trials with different combinations of those to any configuration around and you can see that the the values we said they affect the runtime performance of the spark job quite significantly you can see that the difference is somewhere between 10 and 90 seconds so it's a huge difference based on just configuration so how do we find the best company configuration an engineer might think all right let's try all possible options but considering we have 20 configuration properties each of them can have let's say let's say just 5 possible 5 possible configuration values and a random trial takes around 20 seconds it would take 60 million years to try all possible combinations so we tend to release new versions of our software a bit more often than that although not always right it's also a trivially parallelizable problem so if we run it on 10,000 machines it will just take 6000 years still still not perfect and also it will likely be more expensive because I expect the price of the cost of computation to go down over the next you know million years or so so this is not an option we need to be more intelligent about this one option the most commonly used option is to supervised learning supervised learning is probably the most commonly used area of machine learning where we have labeled data so in our case the features are the configuration values so it's a vector of 20/20 configuration values and the label is the run time the time it took to run that that job and the goal of the supervised learning algorithm is to figure out the function from features to labels and then given a new label it has not seen before it should be able to predict the correct label and if we can do that we can optimize that function and find the best combination of features that produce the lowest possible label if you look at this as a human it's fairly difficult to understand which of these 20 configuration parameters effects the label and in what way but that's what the machine learning algorithms are fairly good at but to be able to imagine it for a human this is a simplified example where we only have one configuration parameter broadcast block size and on the vertical axis we can see the time it takes to run the algorithm and we can see that block broadcast block size affects the run time quite significantly and we can we can fit a machine learning model to represent this function so linear regression probably is not the best fit but even a simple polynomial represents the function reasonably well so we know that the best value that has the lowest run time is probably somewhere between one and eight thousand and we can do the same in a multi-dimensional space here we have two configuration parameters broadcast block size and the number of executor course and we are again trying to find the minimum of that of that function we could either again try a brute-force search and try all possible combinations or use a more intelligent approach such as a gradient Destin so try to find the minimum of that function and here's a tensorflow example of how to do this you know if you don't know tensorflow it allows you to fairly nicely write a code that represents the actual mathematical equations so you can see here that we're defining the neural network and through the weight is a matrix then biases is a vector of numbers and we use X which is the imported the vector of size 20 so these are it's the the configuration values that we feed in our machine learning algorithm and the output of the machine of the of the neural network layer is we multiply the input with weights and biases and run it through a real Lewin activation function and the second layer is exactly the same but for the input we don't use the input to the network but we use output of the first layer as an input to the second layer and the third layer again the same but at the end we don't use the realm function because this is a regression problem so this allows us to produce the output and the training algorithm itself is is fairly simple so we define placeholders so we define a way how we read the data X is the input Y is the output of the neural network and Y with underscore is the expected output so it's the the real label and then we define a gradient destin optimizer so we define how we optimize the neural network and then in a cycle we read batches of examples and we feed those examples to the neural network they produce the neural network predicts an output then we compare the predicted output with the real label and feed the difference back to the neural network so it can improve the the weights the weights and biases and try to learn how to represent and function better and better so over time it should improve and it should learn how to represent that problem fairly well so eventually the network knows how to represent the function from features to labels and therefore we can find minimum of that function and find the best combination of parameters for this particular spark job so problem solved right but what problem did we solve solve the problem were given a lot of data we were able to find the best combination of that data likely if we took the best combination from those random trials it would be very close to the optimal solution anyway so the really really difficult problem is here is where to get the data there are some more interesting solutions and also if what if the spark job changed so if the code changed would the out the algorithm still work probably not so we would have to try to generalize it a little bit but let's talk about a more general case here we have a an example of a large-scale system please don't try to find anything in it it's really just an example but we have rkh DDP that publishes some messages to capcom then we have a car cluster that consumes messages from Kafka and uses Dynamo as an event source it also sometimes triggers a lambda function because why not we also have a microservice has already sketched in a database behind it so the problem with this this system is very complex it's also long-running so we can't really try we can't really run random trials could run some performance tests but that doesn't solve the problem because the system is you know many of your systems you start them and you don't they run forever and they evolve over time they we deploy new versions of the system we change the communication patterns we change serialization and many other things within the system so we would need something that learns over time and evolves with the system it's also a large number of variables each of these systems so Kafka HTTP cockroach DB and all of these they have different configuration values they behave differently they have different failure domains they you know they behave differently under load so there are very different systems and there's there's many variables that we can tune and many of them are unknown there are also many possible actions can for example change configuration of our HTTP or add more Kafka nodes and the reward is often temporal so we don't know if the change we made was good or bad until after a period of time one optional to resolve this is to use reinforcement learning reinforcement learning has been made fairly popular recently with because it was able to beat the the best player in the go game and learn chess and learn other games without actually playing or watching games of other players it just learns the game by itself and it works differently than supervised learning we don't have any labeled data it just learns by trial and error so it we have an agent in an environment the agent takes an action and then it moves to a new state and it receives a reward for its action and by doing this over time it should learn how to behave in that environment it should it should improve its own representation of the environment learn how to behave just imagine the Super Mario game where you have this figure Mario and he can either you know run left run run to the right or jump and if we just you know runs into an enemy or jumps into a hole probably won't receive a lot of reward but if he can reach the end of the level we can reward the agent for doing that and the agent should eventually learn how to reach the end of the level every time in the simplest way how we can represent this agent is by a matrix one of the the common or the quite successful approaches was called Q learning Q learning is represented by a matrix in its simplest form and each of the values in the matrix represents the quality of an action in the given state and when the agent takes the actual action like the best action the action it thinks is the best it receives the reward and it tries to improve the matrix and its own representation of how to behave in each state but it only works in very simple environment with limited number of states and actions so what we want to do instead in a complex environment or we need hierarchical actions and planning and also we have temporal reward such as our distributed system case if we scale up our system that means we'll be paying more money it's more costly so the reward goes down but the system may actually handle larger amount of users if they come so that overall all the long-term reward may increase so the system should learn not only how to gain short-term reward but how to maximize the reward over time so I created this gym in open area terminology gym is a playground where you can you can build your agent against the gym in our case the gym is the actual the actual large scale system it has the concurrent user load which moves from ten thousand twelve to twelve thousand twenty thousand and so on and also the other part of the state is the number of instances in their sizes so one one and two means we have two small instances and one medium-sized instance and the actions the agent can take in this environment is can increase instant size decrease instant size add more instances on or remove instances and the reward is the number of errors so if we cannot handle the user load that decreases the reward and we also have to include cost because if we didn't the agent would very quickly learn to just keep keep increasing the size of the system forever because that would yield the highest reward so we need to include include cost as well so it tries to balance the cost against the number of errors so this is a code of a very simple agent in tensorflow it's only represented by the W which is a matrix of of numbers the agent can pick the highest value from the matrix for the given decision and we also define the trainer so in this case again a gradient destined optimizer and the training of the machine or of the agent is similar as before so for a number of episodes we iterate and in each episode the agent goes through the sequence of states and it chooses an action by picking the highest the highest Q value sometimes it it chooses a random action because otherwise it could just learn to do the same thing over and over again so sometimes it just tries randomly and it takes a step in the environment taking the step yields a new reward and a new state and then we use the bellman equation and calculate the next Q value and calculate the the all Q values variable here and use that to update the agents representation of the state update the matrix of values and by doing this repeatedly the agent should learn how to behave in an environment and should learn how to how to beat or how to maximize the reward so this is a very very simplified State a very simplified solution this is an example of an agent learning how to play the Super Mario game and you can see just like randomly around for a period of time then realizes running to the right is probably a good idea but runs into the first first enemy in the game and dies so and it can keep doing this for hours and hours before it learns that jumping over the enemy is actually a good idea and we don't want our large-scale system to actually do this we want our large-scale system to not not to crash spectacularly it should learn much more efficiently so there are interesting techniques how to how to achieve that or how to at least improve that but eventually the agent should learn how to behave in that environment really well so this is a trained agent and you can see it beats the game actually fairly well it just it runs through the game without pretty much any issues really quickly probably better than most humans could do and reaches the end of the level this is the state we would want to achieve try to maximize maximize the reward and and beat the game so in conclusion still do the you know the optimizations in your code write good code do all the right things about you know delivery message delivery in your system set the right consistency levels validate your system build the resilience in your system just do draw all the good things but at the same time produce data produce observability data from your system and more importantly use the data I'm not asking you to build a reinforcement learning agent but at least look at the data and maybe try to statistically look at some some of the some of the attributes and directions and try to optimize and improve the system in in the chosen directions thank you [Applause] yeah I do I do have some experiments I'm not sure how much of that I can share but we can we can have a chat afterwards so this was a very very basic example of course you know so that the state space search and choosing the the strategy there is is a difficult problem so there's there's quite a few papers in that space you can have a chat chat afterwards I think that's yeah behind the behind this presentation sorry what was the other question [Applause]