sfspark.org: Evan Chan, Configuring and Deploying Apache Spark

sfspark.org: Evan Chan, Configuring and Deploying Apache Spark

Recording: sfspark.org: Evan Chan, Configuring and Deploying Apache Spark

oh all right how did the how many of you guys were at spark summit today all right and you guys have a good time enjoy it yeah all right very good very good um so yeah i'm uh i'm honored to be here i'm invited by alexei to talk about uh production and spark and the spark drop server just a really quick note the picture is uh where i live i'm very excited to find that uh in the audience there's some folks that live really close to me um but uh so i've been using spark for a long time since version 0.8 and also cassandra for a long time since uh 0.6 or so and i have two main uh open source projects that actively uh work on uh one is the smartdrive server we'll talk about tonight there's another area of active research or fun for me which which is working on olap on spark and cassandra if you're interested in that topic feel free to talk to me afterwards as well be very happy to talk to you as well and the uh smartshop server uh we'll go into that in a little bit um but i uh i co-created that with uh kelvin chu who's also in the audience kelvin you wanna race yeah there you go and um kevin's at uber and gave an excellent talk today as well so the spark community is live and just you know really kicking it so deploying spark will start here because really there are so many choices you have um you know do you deploy and yarn meso standalone um you know do you go with one of the other hosted options do you go with the distribution how should you deploy it the funny thing is when i was doing this talk i found there was almost no material on the web or out there that kind of just does an intro and talks about spark um you know production or operations overall like there's some talk you find on yarn and there's some there's probably a talk on my mesosphere but overall it's you know surprisingly little material so so this is more intended you know to get feet wet thinking about overall like you know what uh what things to consider then we're going to talk a little bit about how to uh some some tips for running spark applications and things to watch out for that i found useful and then we're going to talk about the job server uh so first of all like sorry if the words are a little small but um this basic terminology for spark every um spark application is called a driver program or a driver which is on the left side and it creates what is called a spark context and every other operations that you do in spark are driven by the spark context and the way it works is smart context works with a cluster manager a cluster manager then represents uh whether it is yarn or mesos or other things it manages workers it spins them up and it manages the communications it feeds tasks to each worker and each worker node has what is called executors so it's called smart conducts with one or more executors on each node and the executor is actually what carries out the individual parallelized parts of your processing furthermore there is a cache spark is known for in-memory processing and on each executor is where most of the time the cache data or caching of intermediate operations lifts and what the task is you know seems obvious but basically a spark job is broken up to many many many tasks um and so this is just basic terminology is everyone um okay with terminology pretty good all right awesome this walkthrough documentation is really quite good and you can this is this picture is just taken from spark so let's talk about yarn and mesos and stand alone there's a couple of interesting options all of the what are called cluster managers support some common features um all of them will allow you to run your spark apps in a in what is called a cluster mode which means that uh you can actually submit your applications and uh it will run on one of the workers in a cluster and all of them support restarts of your application when it fails at least the most recent versions of spark do um the support of restarting the driver app was i think for standalone mode is relatively recent i'm not sure how long yarn has had it mesos has been able to restart stuff uh for a while through things like marathon and they all have different ui well they all have uis of some sort to expose the state however the uis are different um spark the spark driver actually always spins up an xgp server where you can get the state of each spark application and that will tell you all of the tasks that are running um that is common to every environment and then what happens is that there's a separate ui which is the cluster ui the cluster ui tells you what are the applications that are running and this is different for each environment so the mesos ui looks different than the yarn ui which looks different than the standalone ui um so the spark standard load mode i said it was the easiest clustering mode to deploy although while i was writing this i've you know discovered some neat stuff that might change that but uh the sparks vanilla mode sorry i should step back and explain what it is in case you don't know but basically spark standalone mode means that it runs without something like yarn or mesos you have a master process and the master process is responsible for allocating workers right and um the way um let me see what did i say uh right so um what what the way you package it is that you use the script called make distribution that creates a binary spark distribution you know then you can ship that distribution to each of your workers you pick and then you would pick one of those nodes to act as a master and you would uh run the script called start master and you would typically check that um thanks you would typically check the first thing that i usually do is i tested was spark shell to make sure that it can connect to the master um it might fail and say that there are no workers then you can start some workers then the workers will attach and and you you probably don't shouldn't even need to start restart smart shell the workers because in standard mode you can attach workers um as well so you can add an additional worker it will attach it will register this the master and it will become available however your application when it starts you have to give it a certain number of cpu cores and memory so um that part um is not dynamic so what happens is that the workers if you add additional workers the master knows about them if you start another application then that application can take advantage of those new resources let's see what else is here so there is an x-ray mode for spark master so that if this master goes away another one lives which is done through zookeeper election um now the biggest difference between the standalone mode and something like a yarn ratios is that the standalone mode means you have to dedicate the whole cluster to running spark you're assuming spark is going to assume that you're dedicating all of your cpu all of your memory on each of those workers to spark and yes it is really used in production although actually at all yellow we we actually did run some standalone mode in production so it is possible but since there are other better options you know now let's talk about those other options um so apache mesos uh this is a really brief introduction uh it was started by matthias himself in 2007 before he worked on spark i believe that spark was actually matias idea like an an idea of using meso so mesos was actually his research project and spark was an application to run on top of mesos what is very ironic is that since then um mesos hasn't necessarily been the most emphasized cluster manager in spark but yeah that's a very interesting history mesos is not designed just for running your big data or hadoop workloads it is designed to actually run all of your applications with your company so an example is that for someone current company uses mesos to deploy microservices so you can deploy docker containers on mesos and there they have mesos has multiple resource managers that can work with microservices and restart them the most important one is called marathon but there are other ones out there twitter has one which name of which i forgot anyone remember i don't quite remember what it is but um so so marathon will actually deploy your apps and it will know how many instances there should be if one fails it will start up another one it will also do dns routing so that it can route actually requests that go to one endpoint to two-year services for you and mesos can also run non-jvm workloads because it uses a c plus libmesos.so library so it can work with a lot of workloads like mpi the downside of of this is of course that you need to install the mesos and run jni however i've heard that i've heard from mesosphere that they are working on an http api so that you will no longer need to install the mesos which would be pretty cool it is heavily used at twitter and airbnb and mesosphere is doing a lot of cool stuff at this point i'll call out the mesosphere dcos which if you don't know how many people have heard of mesosphere decals you know it's very very exciting it will let you deploy spark by typing in decals package install spark then you install cassandra you just do the cost package install cassandra so i'm just waiting to check this out this sounds very very cool a little bit about mesos versus yarn um yarn is a resource manager and a scheduler mesos is a two level resource manager which basically lets resource managers schedule stuff themselves so this is kind of this seems like it's a very important distinction um basically you can maybe one way to explain is that there's a project where you can run yarn within mesos and the way that works is that urine will let mesos delegate or offer resources if you run you are normally in hadoop the yarn will also know about all the resources if you run down mesos basically mesos at the top level knows about all the resources and it lets other schedulers then receive basically mesos will offer resources to different schedulers and the schedulers will then take those offers and decide how it wants to schedule them so in this case mesos will offer resources to say yarn it can also offer resources to say marathon and that way yarn and marathon can cooperate and manage your whole your entire data center and offer some resources to to yarn for running regulator workloads and then offer some resources to marathon for running your docker microservices you can also write your own schedulers within mesos i guess this is probably obvious but if if you are a hadoop shop um there is obviously a strong incentive to use yarn because you're probably upgrading your cluster to use hadoop 2.x if you're starting out i would recommend that you go with mesos because you can run your whole infrastructure for your company your mesos offers two ways of sorry spark runs mesos jobs in two modes which is very interesting one is called coarse grained and one is called fine grain what core screen does is it's very similar to the spark standalone mode at the beginning of your spark application you give it a fixed number of resources and mesos will give you a fixed number of workers and that number stays fixed for the duration of your application example if you want to run a bunch of quick fast queries um each of those might consist of a few tasks and then you might run another query that has a few tasks you don't want to be allocating new workers every time you get a new task so in that case having fixed resources is probably a good thing fine grain which is actually default means that you have dynamic worker allocation so with every task it will actually go back to mesos and ask for resources this means that as your job is running the size of the worker pool will change but it also means that every task has a certain startup latency because it has to wait for mesos to allocate resources workers so but it is very interesting and i think we've tried both um what about data stacks just a raise of hands how many people are running spark on cassandra a few folks cool i just wanted to mention this data stacks has their own implementation of spark on cassandra which is very interesting it packages cassandra hadoop and spark on one so um it's a different resource manager what they do is they use cassandra gossip um as an actual mechanism for spark masters so it's like standalone but with cassandra coordinating the failover instead of zookeeper it's pretty interesting if you use data stacks dse you can use the cfs which is data stacks implementation of hdfs on cassandra and you can use that to yeah to write your files or you can just write data to center i also mentioned tachyon here which is uh you know pretty interesting option to think about uh it doesn't just apply to cassandra applies to everyone but um if you are say not using uh data stacks enterprise you could use tachyon as a caching layer between spark and cassandra and then that would work with things like mesosphere where you at least as i understand right now you can't really guarantee that you can co-locate your spark and cassandra clusters um you guys might might have heard so what about hosted spark right so you guys might have heard the announcement today that amazon is making spark a first class citizen for emr which is pretty cool i'm not sure what the difference is exactly to be honest but before that they were running spark on top of hadoop 2.0 yarn and i think some people complain about various problems configure for configuring it so hopefully this will be you know much easier now and would be a pretty compelling option um google google compute engine apparently has a click to deploy hadoop plus spark anybody run on google compute engine here no okay um there's obviously databricks uh i guess it's no longer database cloud it's just databricks so um strike that and i suspect that many more are coming like if you went to the summit you noticed that there were various vendors that were offering uh hosted spark well i know of at least one so this will be a very interesting space to watch anybody care to share other ones that they know of like other hostel solutions okay oh yeah i guess azure might yeah azure is probably do not yeah blue data claims oh who's oh blue data okay blue data for i think that's the vendor that i saw yeah that has hosted spark so this would be a very interesting space to watch um this is an interesting note about the different environments every environment has a different way of starting spark dsc has dse spark or dsc sparks on it emr has its own command dcos has their own command for writing applications so this would be a very interesting space you know how do you you know unify all these different commands maybe you can use a red service but we're not quite there yet so a few tips on configuring spark make sure you build it for the right hadoop version if that matters building is kind of a big chore if whenever you need to um install spark today um i think there are binary versions of like i mean if you get it if you get spark through distribution then obviously this is taken care of for you but for me i've always had to build spark um and you can also build it for different scala versions uh which is something to note um they offer for both spark 2.10 and sorry scala 2.10 and 2.11 the spark job server also lets you build for 2.10 and 2.11 make sure your scala versions match jars our favorite topic basically uh dependency management on jbm is a royal pain there is really no other way to put it every distraught slightly different things if you run cdx how many people here run cdh spark feel you have to be aware that cdh before 5.4 shipped with a different version of akka from what the regular spark distribution did that causes pain if you use it with aca and spark which the job server does for example um i highly recommend leaving out hive if you don't need it because hive brings in like about you know double the number of dependencies there is a ui that i'll show you later there's a tab which is really convenient for checking your dependencies and where it came from the i'll mention this again later but spark summit has really useful flag called dash chart or death packages which forwards your jars to each worker um however you might want to actually use a spark class path and push your jars to each node the reason is because you might not if you especially if your big jar you might not want spark to download um from your uh driver um to endnotes however big ns every time you start a new job and the behavior at least is of 1.3.0 is that when you start a context that's not when spark downloads the jars the smart downloads when you start your first job so that whatever the first job is is going to take a hit that first time um so some useful config options if you if you do a lot of network io or shuffles you'll probably want to use the creo serializer the problem using creo is that you have to register every class every custom class that your application needs spark does include all of the common java collections scholar collections and other common scholar classes but you do have to do some work but it's pretty much always worth it if you need performance the default parallelism for controlling the number of shuffle reduced tasks this is useful for certain times when like a lot of people don't use the javascale dsl right they use other frameworks on top so that's a way of controlling the there's a default parallelism level that gets used for the reduced tasks which you might want to tweak um this is a schedule let me see the memory fraction is and the storage fraction is important if you want to cache your rdds if you have a storage memory fraction of 0.6 that and you have let's say i can't really do math let's say you have 10 gigs of heap right so then that means that spark will use at most six gigs to cache your rdds and you have four gigs left over for everything else including processing and if you run long running contacts you will care about spark cleaner ttl which is tells you how long spark is going to wait until it cleans up rdds that are cached this is usually set for streaming jobs for example to make sure that there aren't too many outstanding rdds let's see what else there's a max failures the scheduler mode won't be relevant for most people but it will be relevant for things like job server i'll mention this again later but it enables you to do some level parallelism for tasks if you run spark sql you'll probably want to set spark sql shuffle partitions unfortunately it is set manually i believe that they're working on some kind of auto some automatic way of figuring out number of partitions but for now you need to set ahead of time how many shuffle partitions that you need and um my let's see do i have something else for this um oh i guess not i was i was going to say like you you should um i think my rule of thumb is is to oh maybe that's some for something else usually for stuff like that it's like that for the shuffles you know what no i think that's something else okay never mind um there there is a there's various policies for controlling the number of temp files if you're running in standalone mode you might notice that your disk will fill up and this is because spark by default will not clean up your temp files um half the things i talk about are just things i know about because i either contributed them or commented them or that kind of stuff so it was pretty easy to run a crown job yourself but it's good to be able to set a flag for spark to clean it up running spark applications so there's different ways to run your apps in the cluster you you definitely want to avoid running jobs on your local machine we'll talk about how the job server helps that um and there is um if you run uh wait a minute sorry uh so yeah so the deploy mode cluster means that it's running your um your application in the cluster itself as opposed to say on your laptop or wherever you submit it from and i'll just note that if you want so this is an important feature if you want your jobs to restart when it fails you can use there's different ways well every like different way of running spark has its own way of doing this um you can do deploy mode cluster for standalone mode you need a supervised flag for it to work for mesos you will get that automatically if you use marathon and one thing that is good to note for these restarts is checkpointing because spark by default you know it will recompute your entire chain of computations and you can use checkpointing to help shortcut that um let's see oops so right so speeding up application startup i think i mentioned most of this already but if you want to you want to avoid the packages option if you want your jobs to start quickly because otherwise it will use http broadcast to basically it will download the jars from your driver's hdb server which might take a while um if you want really fast for our jobs we can talk about the job server and how that enables that use case in just a minute um you can get metrics so when your productionized spark metrics is probably you probably care a lot about metrics so there's a built-in metric system that has source and sinks and it's all written based on the koda hail or yammer metrics library so the sinks are the same as what koda hill offers so graphite ganglia et cetera et cetera and there is a json metrics thing as well so there's a metrics.properties file that you can configure to configure the syncs and sources um i will also just you know you guys can click on this link well i won't click on it now but um for you know ways that examples of visualizing basically there's people that build say grafina dashboards where you do visualize the built-in spark metrics which is pretty cool and my old colleague at oyala also builds an interesting small library on top of spark accumulators that ties that into so how many people know what spark accumulators are a few people so let me explain so basically let's say so in hadoop you had counters so spark doesn't have counters so what do you do right so spark has something else called accumulators which is where on each task you can if you have an accumulator then you can add some number to it and then spark will automatically get all those counters and merge them at the driver and you end up with the sum from your job so that's how you implement counters in spark there's a library there that you know you get the slides later but that would tie spark accumulators in with the spark metric system that way you can use accumulators and the the metrics from the application would then get fed to all of your syncs such as graphite et cetera so that might be a good way for you to get application metrics out i don't there's anybody aware of other ways of getting metrics out from spark of patients but you want to watch how rdds are cached there's a really useful storage ui tab that tells you all of your cache rdds and where they're being stored if you see this size on disk is non-zero and the size of memory is zero that you know would probably tell you why your cache queries are running slowly if you're if your jobs are stuck um the um there's an executor's pane uh of your spark job and that now has a thread down link which is pretty useful you can click on that and you know get a thread that i'm assuming that the ui is responsive finally this is what i was talking about earlier there's actually a very useful under the environment tab there's a whole section for class path entries the ones that say added by user are the ones that are added with the dash dash packages option the reason why the packages option is useful by the way is because it will actually pull down dependencies from maven so for example let's say i run cassandra i want this cassandra spark connector i can add maven coordinates and it will download that one and all the dependencies which is you know super convenient but again it will you know if don't do this in production because in in production you don't want like your spark thing to be downloading for maven you know so um yep so the spark job server we come to um why um so just a really brief of history or sorry an overview of what the job server is it is basically a rest api for operating spark jobs and running your spark cluster so you can submit jobs to spark through the job server you can see with your status and you can manage spark applications one particular use case so so we um one particle use case that it's good at is for sharing let's say that you want to share rdds um and so in no regular spark each executable each application it's its own entity spark job server creates a separate abstraction called a job and a job is actually different than a spark drive or application the notion is that you can actually run multiple jobs within the same spark contacts or driver and and that lets you do a lot of neat things such as i can have different jobs be low latency queries or i can basically modularize my spark application instead of having one giant application or driver i can create separate components i can test them independently so i think it facilitates a a better way of developing spark apps it will also save all of your jars and the job history and configurations to a database and so basically you can go back and you can revisit you know history and things like that and it has an uh a and async and api it takes your job results and converts it to json so again you can interact with it from you know virtually any language and it has support for spark sql for hive contacts and basically it can be extended to support virtually any type of spark context and it's open source you can find it at the url and you can also find it at spark packages so just a really brief history we created kevin and i created together at uyala at the beginning of 2013 when it was just spark 0.8 this was before it was spark submit this was before there was a spark history server and all kinds of stuff and the reason why we needed it is because spark at that time the way that you ran spark applications was that you just ran the application uh on whatever on on your local box or on some typically in some cluster and we really wanted to make this easy with you know like for us we envisioned spark as a service that multiple teams can run um we didn't want to create what was happening was we would have to set up batching boxes on on a cluster like say one per team or one per person or something and to let them run their smart calculations and that wasn't really a scalable way or a secure way to let teams use spark right so we wanted to um have people focus on uh writing the jobs and not on deploying them and tracking your job status and this kind of thing so we want to provide a common infrastructure to operate spark as a service by developers that want to run spark jobs and finally like we had a pretty polyglot technology stack with ruby we had go so this is you know another reason why we wanted it so i'll go over an example workflow this is for a scholar project that you would add a dependency on the api and you would basically create an assembly jar which excludes spark and other dependencies and and then you would upload it so a really quick example of a java is that basically you have to implement an interface there's two methods in interface it's relatively simple one is run job which does exactly what it says it does you're past a spark context and that's a very important difference you don't create the context in your app it is created for you and you're passing config which is basically passed in when you do a post and there's a validate method the reason why there's a validate method is because this is following http semantics uh when you when your input is not valid uh you don't want to just you know run the job and fail with the 500 you want to return a you know 403 40x you know code to indicate that there's a problem and the response so this is to facilitate a standard restful api and it also gives you a chance to fail early from your job and not get some random unhelpful exception notice that i can actually specify a reason why the configuration data input is not valid right and the run job itself is it does exactly what you would expect you know i can do the standard spark commands to do transformations and the response that the result of that function would be then automatically serialized and returned to the uh to the user so what is different so i said this the job does not create a spark context the job server does and this allows you to decide exactly how you want to you know run the job like you can you can pre-create a context with certain parameters and then you can run the job so it separates out the uh the creation of the spark context and and all the things with the the actual job functionality um it allows for modular spark development like an example is let's say i'm working with spark sql on cassandra or some other source i can have one job that it's that is just for loading um the tables and that can be tested independently then i can have one job that is just for currying them and again that can be tested independently and i can have one job to do diagnostics for example i can load it it can figure out what tables are cached and you know tell me other things and report debug information this is a little hard to see but the top studio one curl command that would upload the jar to the job server and you specify a an application name which in this case i call demo the job server tracks different versions of the jar so in the nyx curl i'm actually giving an input in this case it is in line in the curl but more typically it would be uh referring to you know whatever rest api you use you know you would be it would be it could be a json blob or it could be in the type save configuration format hokan you know json is completely fine and in this case my job jar is called word count example and i'm just going to take that sentence and count up the words and you can see that i get this result which is a json map of the words and accounts i can very easily retrieve the job status let's say i want the last two jobs that ran and i can see how long they ran and what the clock what job was running and in what context and what happened so if there was an error you would see instead of finish you would see error and what the exception was and since then we've also added logging of our saving of the con of the config so basically every job input that is passed in is also saved so you can go back and easily trace what what was the input that caused the error so very quickly we go into fast query jobs the the goal of this was to have spark jobs that can run very quickly in um you know a second time frame and answer queries on shared rde so this is using spark as a query engine as a rest query engine the query params are passed in as the job config as json and we wanted we wanted to minimize the so um the reason why we don't just submit a regular spark app is because we want this to be low latency when you start a spark application driver it typically takes at least five seconds just for to spin up all the threads and the http server and the file server and all the other things that spark needs so starting a new application is relatively heavyweight and that's not counting all the time that you're typically running this production mesos or yarn would need to allocate all of your working processes right so so this way we can have individual logical jobs that are separate from the application and they can run very quickly so the way that this works is that we have a rest address job server we would the first step would be we create a a context a spark context and so this creates a context which spins up the executors next we can run some job that will load some data so this might for example say get the data from cassandra or sell elsewhere and it would cache the data as cache rdds and the cache rdds stay persistent in the spark context after the logical job returns at which point i can run a query job which loads tasks that basically inquire into the the saved rdds cache rdds and returns results and i can do that again and again um the job server provides a functionality called named rdds where you can name what rdds to save this is pretty important because if you don't name them spark only gives you an api for a numeric id and so it's fairly difficult to figure out what rdds have in cache so it gives you a way to name space you save rds and it is also designed to be thread safe so that you can run concurrent logical jobs or tasks let me skip this a little bit so here's an example of a spark sql or hive query server um again so i start a context based on sql context notice that there's a factory so basically you can implement a factory to start up any kind of contacts you want um whether it be the built-in spark ones or you can create for example a custom spark contacts type that holds some business specific data and then you can interact with that in your job then i would run a job for loading and caching tables in the data frames um and then it will supply a query and the query could be sql there actually is a built-in sql query job in the job server and this was a use case that we just got which is pretty interesting someone wanted to combine streaming and uh and spark sql so the way it works is we create a special type of context called a spark sql streaming context this is to come into job server but i'll just show you the code is actually fairly small so let's say that you have data coming in from kafka and you have what you would have a streaming job that is running all the time the streaming job would create data frames and those data frames will be saved in as the sql context tables then you can run a sql query job on top which is querying those saved data frames so pretty neat the code for doing this is pretty lightweight i define a trade spark sql streaming job and i say that those kind of jobs need to be run under a sql streaming context the sql streaming conducts is basically a wrapper around smart conduct that starts up both the streaming context and the sql context so it's pretty self-explanatory jobs would then take in a sql streaming context and have access to both the streaming parts and the sql contacts this is a little hard to see so but basically there's a sql streaming contact factory which knows how to create a sql streaming context and taken configuration parameters an example here is this actually came from someone that uses a job server which is pretty neat but um you would pull out the batch interval for spark streaming from the config right and then this will be used to initialize it so that way you can use put do a rest call that initializes this kind of context with your batch interval and it would know how to stop it and start it etc so just really quickly um the spark job server uses a metadata store to store all the statuses and configs and everything else right now it is defaults to an x2 database you can configure to point at any jdbc url and multiple job servers can share and actually is recommended to share the same database for xj that way you can share jars and everything else right it also comes with scripts of deploying as well as routes for getting metrics out there are some challenges i think that you know originally basically we deal with a lot of class path stuff and dynamic loading of jars because it has to load jobs as jars so that's an additional challenge that that we had also uh when we started this in the early days of spark there was no restriction on running um multiple smart contacts within the same jvm but now there is so you need to set the special flag allow multiple contexts equals true this is a short-term solution though so the long-term solution is that we're going to fork off a separate jvm per context and finally some interesting future work that is coming up um probably the most requested you know things are uh true xj operation which means i'll explain what that means in a second um and things like user permissions you obviously don't want to allow anyone to delete jobs or you know that kind of thing so that would be pretty interesting we'll be adding that pending time and everything else just a really quick some slides on what uh xa you know means an example what i mean by the true xj is for example you can have let's say you have two job servers and one active spark contacts that's running if one of the job servers dies then your load balancer would reroute the request to job server two and because they share a database and because the job server 2 knows about what's going on through gossip it would know that it can then find out status from the active job so it would keep that connection the connection to the context would fail over to the job server so that it doesn't matter which one you ask they would they would all know they would all have consistent state so this is interesting even more interesting is we could potentially have uh h and heart failover for jobs themselves um this this is a little bit of a this is just a design so it's all welcome to comment but imagine that you have an active job running this is producing home state it's checkpointing files to hdfs right now what happens is if it fails yarn or mesos could restart it then it's going to load that checkpoint file if you've enabled it but imagine if we could have some kind of heart failover right so so that you have a standby spark driver that is actually running and it is able to continuously reload state and update the checkpointing that way when it fails then that way it's no longer just a restart but the job server can enable a hot failover of spark jobs where there is very little downtime you know close to none because the state the expensive part which is restoring application state has already been done obviously this won't work for all applications but you know welcome to hear your feedback i think would be pretty cool and you know we'd like to thank everyone for the contributions we've gotten a lot of community contributions which is really awesome and you know we definitely welcome more contributions i'll skip this part really quickly but there are a lot of documentation for how the orchestra works thank you very much i think there are some and i think there's um at least one in the audience so you can you can talk to how's that um uh sorry so you said how reliable when the task failure happens right how how reliable is the value of an accumulator yeah i'm not really sure that's it i think that's that that's a that's a good point because i think the updates get sent to the driver you're talking about if just not the driver but you know individual executor you know test fails um and i'm not sure that spark knows how to roll that you know stay back but um yeah that's a good question yeah um people watching like can you go back to the architecture slides oh sure oh okay all right sure no problem um yeah is that is that good okay architecture one two three we'll let some good music yeah right that sounds good sounds good all right there you go thanks yeah um available for questions afterwards too so yeah available later yeah yeah they'll definitely be available yeah all right thanks very much we'll welcome you