data.bythebay.io: Ilya Ganelin, The Engineer's Guide to Streaming
Recording: data.bythebay.io: Ilya Ganelin, The Engineer's Guide to Streaming
hello it's late in the day I'll keep it a short sweet hopefully leave lots of space for questions today I i really want to dive into kind of a big problem i run into over the past you know six months to a year of working with streaming platforms in that they're all essentially the same right like in the open source world you have 12 / 13 or 14 frameworks now all at first glance look fairly identical and it's actually kind of hard to distinguish what they're really doing under the hood until you really dive into it and understand it and the reason this matters is when you're building a product right when you have a concrete problem that you want to solve you're going to have a unique set of requirements and each one of these streaming frameworks that's out there solves a particular problem for the people that build it and whether it solves the problem that you have at the present state of time right that's kind of an open question so my intent with this talk is to kind of highlight how I think these frameworks actually distinguish themselves and in kind of the engineering world where you need to build a system that will run that will be stable and where you've kind of considered the whole of what you're building that there's really a few key things that you can look at to differentiate these so a minor minor addendum data innovation lab a capital one also known as vault eight we're essentially an R&D team we go out play with all the new tech about heavily with open source and then ultimately bring it back to tackle concrete projects within the business so that's kind of the genesis of all this investigation understanding what have you so firstly kind of very quickly we're all here big data folks I'm not going to spend a ton of time on this but there's kind of a fundamental distinction between batch and streaming and and really the big thing in my mind is is that how these things are built around is is really a function of latency they both are going to process at large amounts of data as we call it but the time scales that you're dealing with in streaming systems are going to be on the order of milliseconds in microseconds rather than seconds minutes and hours and this is really in my mind what distinguished true streaming coliforms from things like spark streaming where they micro match and under the hood and really you're dealing with a batch platform even if it says it's a streaming platform so quick examples MapReduce Park tez on the batch side storm flank apex kafka streams gear pump concord aqua streams you know the pick here pick your run-of-the-mill we had you know anyone presented just now heron as well but not open source yet uh four key things in my mind that really do matter when you're talking about these things and and there's kind of fundamental questions that we all ask and it's how do we handle more data how do we handle that data quickly without ever losing data or compute and this one is huge right like how these frameworks solve this particular question turns out to be a key differentiator and then kind of a slight nuance on that how do we show these systems keep working even if there are failures and these are these four concepts right scalability performance durability and availability in each one of these has concrete metrics associated with it for scalability we're never going to get true linear scaling the idea is right we can add 10 more nodes to you know to double our cluster size we're not going to see a 2x increase in performance we might see one and a half X if we're lucky as close as we are to linear that's the best that we can hope for for performance I'm talking about throughput and latency right these are these are hard numbers for durability I'm talking about loss of data and loss of computer can messages be lost can can we lose state these are kind of key questions and lastly availability is for everything that comes in for every request that the system receives will it generate a response and this turns out to be kind of a key differentiator when you go from the you know Toyland of prototypes and fun problems and exploration into actually building a product that works and especially a product that works at scale nothing is that all of these are kind of intimately connected there is you can't look at any one of these in isolation when you're either analyzing a system or when you're building a system and there's that's because there's really these trade-offs right and concretely if you focus on scalable right making systems that will run larger and larger and larger then first of all you're going to have challenges with durability the more moving pieces you have in your system the more room there is for something to break you're going to struggle with availability because there's more things to keep track of if you have more things breaking that means you have to be more resilient to failures you're going to struggle with performance because when you have more components of your system you're going to have bottlenecks you're going to have challenges with synchronization consistency you name it on the flip side if we focus on availability then performance is going to be harder to guarantee when you want to ensure that your entire system stays operational you're going to be dealing with the overhead of monitoring and synchronization you're going to have challenges with scalability because again these are kind of contradictory concepts in some sets and you're going to have challenges with durability because recovering without losing data means that your system has to be even more robust and lastly durability kind of suffers for the same all the above results now in the batch compute land they have it easy right you get scale out in performance because you can add hardware and you can simply take longer because the time scales that we're dealing with are different we can actually most of the time just get around the challenges of scale out by throwing more hardware at the problem and that was kind of what happened with MapReduce what have them sparked right the bigger your cluster is things just magically start working you get durability when you use a stateful backing store map produced in spark there ultimately based on top of HDFS or s3 you know they're there on disk they're on a physical disk if you cut power that data is still going to be there that data is still going to be replicated in a streaming system you don't have that luxury you can't have this recompute because that data is in flight it's never landed you get availability because the time scales that you're operating on our longer if you're running on yarn yarn will restart a process in 30 seconds in batch compute land that's a very small amount of time in the streaming world that's an eternity so kind of the the crux of this is in the land of stream processing you don't have this time you don't have this luxury of able to wait for something to recover wait for your system to read data back from its durable store or your Kafka to get back to an operational state so it's really not about performance and scale if you look at all of the streaming framers out there most of them are going to have numbers akin to they're going to process a million transactions per second per node or they will have Layton sees on the order of tens of milliseconds right that's not what sets them apart and these things will run on thousand node clusters storm for example does does great but the challenges that you run into our kind of these four things that I laid out here is I see the first is ease of use and when I say ease of use I literally mean how quickly can you build an application that's more than a toy problem I don't care about word count word count doesn't help you solve a real challenge I'm saying a legitimate production application with all of the intricacies associated with actually running something at scale in a distributed environment to get to the point where you can legitimately hand it off and say hey guys I have a product it will work if things break it will recover how easily your framework provides you the ability to do that turns out to be a quickie differentiated bring us a second point and that's failure handling what happens when things break in a distributed land things are going to break so what does your system do when that happens does it handle failures by sacrificing something else right can your system be resilient to failures without sacrificing throughput or latency or availability durability how do you avoid losing data without sacrificing performance same idea and for availability this is really kind of getting into DevOps land how do you keep the system running with a minimum of manual effort because at the end of the day there will be a person that has to restart a failed process or restart a failed node and the more people that you have in the loop the more room there is for error the more room there is for this process to take longer and for things to never recover fully because things like the cascading failure if you have a system that recovers on its own you have a much more robust solution so next I want to just look at three particular technologies and these aren't selected for any particular reason aside from the fact that I'm very familiar with these and I think that they highlight a lot of kind of the key issues associated with streaming technology in general right again I spoke to kind of a lot of these things being similar many of the concepts I'm going to talk about today will generalize you'll see the same things when you look at how kafka stream works how aqua streams works how gear pump works there's a lot of similarities there so we'll look concretely it's storm fling kinetics and look at how they deal with these four things that I listed so first we've got Apache storm and apache storm has been kind of our dinosaur it's been around forever works really well Twitter ran it forever on massive clusters it was scalable right 10,000 nodes it was performant you could run storm fairly easily at millisecond single millisecond latency and it was easy to use you can you can stand up a storm cluster on a machine or on you know distributed environment fairly quickly you can write applications in a fairly straightforward manner once you wrap your head around the idea of creating topologies for passing around data but it had a number of weaknesses if you if you were Karthik's talked earlier on an era he dived into some of these but essentially failure handling compromises performance and it compromises it in a very kind of drastic way I'll dive into that a second the second bit is that keeping things running at scale in a production environment turned out to be a major challenge there were you know a number of challenges around resource sharing scheduling reliability dynamic application deployment things that you need when you want to keep something running over a long period of time and the last bit is kind of flexibility which ties into this notion of building dynamic reconfigurable applications that are easy to modify and easy to adjust to whatever use cases you right into the last bits kind of an open question I'm not going to stay up here and say it there is a lot more noise in the Big Data community about most other technologies you don't hear much about storm anymore storm is still running in many companies there was at the same time a major patch that just came out for for someone dotto that added a lot of features that have been missing over a while and Hortonworks is definitely still selling it very hard as it's streaming platform of choice so I'll leave that up to you so in storm uh very simply most of you have probably seen this in one form or another you've got spouts that emit data you have bolts that consume data they pass data between each other bolts are logical operations that transform your data in some fashion and admit it to downstream consumers and you can obviously have multi-input multi-output connections concretely you've got kind of different ways that you can pass around data these groupings aren't super important essentially this is just getting it the idea that you can do a lot of things in parallel and storm you achieve scalability and parallelism by applying the same logical operation to different physical pieces of data so if you have a vector of data that comes in that vector of data will be physically partitioned to multiple nodes doing the exact same thing to the data and then passing it downstream now what you wind up with is is a graph ultimately that looks like what you have on the right data is passed back and forth between nodes in storm these nodes are actually separate often separate processes this is not a configurable things normal kind of deploy operators and bolts and spouts where it has the resources to do so when we talk about apex later we'll see how being able to configure this programmatically can actually help you build more robust applications so what happens when you have a failure in storm is that the upstream provider the spout will emit data and that will basically wait for a response from downstream saying hey I've got the data I just it and the only way that you can do this is if you have a heartbeat right you're essentially saying hey are you done yet are you done yet are you done yet at some point you timeout say well I didn't get a response well what am I going to do the problem is that there's no way to do this in storm any quicker than one side there's kind of a hard limit on a minimum act time and you can get around this problem if you kind of say hey well okay i'm gonna i'm going to paralyze my system i'm going to do the same thing in multiple parallel pipelines all doing the same operations to the data but that way if one of my pipelines fails the other two will still succeed and I'm guaranteed to at least have one output at the end and if I'm okay with at least once processing or I have something downstream they can take the first response out and say hey I'm done this is an acceptable solution now the problem is that in storm when you have this failure if you have a bolt that fails downstream promised errors propagate the only way the storm can reset its state and recover and replay what it ran into in the case of a failure is to actually reset the spout and resend that message again and the problem is that this resets the entire topology now this issue of cascading failures makes it harder to create a program like what I just described where you have these independent pipelines what this means is that instead of building a single application that can deal with failures the only way that you can get around this is by deploying multiple applications but then you run into the kind of the complementary issue that storms not very good at running multiple applications simultaneously at scale this is kind of diving into the fundamental challenges that Twitter ran into a storm and why they built here on so it's kind of you know a chicken in the egg problem and there's a number of other weaknesses with storm the the big one is you know this this recompute on the entire tree it's hard to debug because operations in individual operators are shared right you might have multiple logical operations within the same jvm and that means that you can't isolate individual processes a big big push within hair on was to actually separate everything into individual tasks and processes so that you could see and pinpoint where anything failed and also to again isolate failures so that they don't affect other components and the last bit is is this kind of getting into this notion of dynamism right if you want a living breathing environment that you can change and adjust to solve your needs then you want to be able to have dynamic topologies that you can modify and reconfigure in flight you want to have dynamic scaling where you can increase the parallelism of your system to handle more data and you want to be able to add and remove applications without service interruption and storm does not have many of these things even with the recent one dead o release so let's look at a second example let's talk about flank and in flink has a really position itself until very recently as a streaming competitor but now they're kind of pushing on this very hard but a long time they were kind of ganging up on spark and this turned out to be a really brilliant move because everyone was talking about spark and as soon as flink came along and said hey guys I'm the next barik everyone started talking about flank and blink did one thing super well and that's that they started with a declarative high level API right the one thing that makes spark most successful in my mind is that it's easy to use infinitely took the same approach it's got the same notions of MapReduce filter join group by and you write applications really simply under the hood flink will make the same kind of topology that you are manually configuring in storm but what this means is that you're kind of time to function application is a lot lower now we're also brings to the table is like storm it's a true streaming platform right is legitimately processing data one piece at a time it's not doing micro batching under the hood technically it is but that's for the sake of network efficiency and it's a configurable thing and the guys did a really brilliant thing in the the the folks that built flink were actually database engineers and so they built in a really kind of powerful query planner that goes a step further than what spark does where they kind of collapsed the dag as much as they can to minimize disk reads they will actually compute the optimal dag for a given set of operations that you present making it easier to build more efficient applications and the last bit that's kind of really in favor is this notion of community right so if link has a very fast growing presence in amongst meetups amongst conferences and that turns out to help a lot primarily because it gets more people involved in more people contributing and open-source project succeed based on how well the interface with each other and how much excitement activity there is around them this kind of whole environment really depends on the feedback and interaction that people have with each other so succeeding there makes it much more likely for your project to succeed in the long term so to that point right how does how does link work same same basic idea a storm except you're no longer talking about individually configuring where data flows in your system you're actually just saying what you want to do to the data and flink figures out how to transform that into a graph now the next bit that's interesting is how flink handles failures instead of the storm approach where you have to have something upstream produce data and then wait for a response to come along that it can acknowledge as having been processed what flink does is they inject these markers into the streaming data these are treated like any other piece of data with the exception that any time an operator sees the barrier it says hey okay I've processed all the barriers that came before me I'm going to save my state thankfully does this in a distributed fashion so every operator will write to distribute a store lying underneath and it will do so independently what this means is that flink can both serialize its state and then recover its state really quickly without the additional overhead of having to wait for a problem to come along and the most that it ever has to roll back is is to the previous barrier and these can come along and configurable intervals now the the advantage of this is that if you kind of write sensible applications that don't abuse the amount of memory that you're storing the amount of data that you're storing in the memory Beach operator and then there's not a lot of state to serialize this is a really lightweight process and they have some excellent benchmarks in their blogs showing this comparison with storm of even in failure scenarios we still keep running with more or less the same throughput and the same latency as we would in a scenario where we're just dropping failures on the floor and not recovering and you'll see that storm will do terribly right like storm will have an immediate massive performance degradation wear a splint will just keep running fine now more on kind of the differences with storm as if link handles kind of planning an optimization for you you don't have to come up with your optimal dag the lower level internals are abstracted which is coming back to this ease-of-use idea right how do I quickly write complex applications that just work another huge thing is that flink has clear semantics around windowing if if folks have been following kind of the discussion around beam which is the notion of building a unified kind of API around streaming platforms this is one of the key things that you have to have how do you deal with out a board of data out of order data how do you know when all the data you wanted to process has arrived link has that explicitly defined and they're actually the first project open source project outside of Google that has actually implemented a fully beam compatible or mostly be incompatible implementation for a streaming platform i should say failure handlings light weight fast and another big thing is that in the storm world because of the this acting mechanism there's actually no way to guarantee exactly once processing because the downstream operator might have succeeded but it might have just taken too long so the only way in storm to guarantee use after once processing is to use the trident framework which essentially forces a micro batching on top of the underlying storm API in flank you can get in exactly once guarantee as long as you have a conductor that deals with a stateful database or data store upstream they can actually do this kind of act response and says hey I'm going to I'm going to have written everything up to this checkpoint and the database can say that's right you've done that so something akin to a Kafka right where you know how far you've consumed in your buffer except just on the output side and kind of this very very nice thing at the end is if link will actually run storm applications natively which means that if you have storm topologies already written then you can migrate definitely transparently and it'll just work so what can't flink do think still cannot dynamically update as topologies and it also can't dynamically scale out to deal with more data these are so kind of static things the other big thing is that Flint cannot recover from errors without stopping the entire dag in the same way that storm had to roll back all the way to the source in order to recover from failures flink has to do the same thing think does it faster and more efficiently but it still depends on an upstream durable store from which to recover something like a Kafka or a you know stateful repository on HDFS in order to recover its state and replay all the messages that were not processed from the last received barrier it also can't give you kind of fine-grained control over how data moves through the system and when we talk about apex we'll see why this matters but the thing is that this high level API right this ease of use this isn't always a good thing it's kind of a it's kind of like a hidden curse in some sense and spark runs into the exact same thing that on paper spark is super easy to use it's it's easy to write you know complex etl pipelines in spark until things break and when things break then you really have to dive in and try to figure out hey why did things break and then this high level API that made it really easy to write this application turns out to be something that just gets in your way because you can't decompose it and actually fix the things that you want to have control over and the last kind of question with link is is really can this thing run in production very recently have they they first started they started to announce their their first production customers and this really matters ultimately when you're trying to come to the business and say hey I have this new technology I want to build critical infrastructure on this technology and you want to be able to point something else and say these guys did it right that's a huge selling point for storm storm broadens their our production environments running storm there are applications on top of storm that run reliably and stabili you can point to those and say hey they did it so can we what's link you kind of can't do that yet and this makes it harder to kind of get buy-in for whatever you're buying know for whatever you're building and you can't do anything alone so there's a lot of other things out there there there was a blog that came out about a month ago guy basically printed out a chart of open-source streaming technologies and compared them against a number of criteria he missed a bunch and I think three or four more came out since he did that chart a month ago so there's a lot of competition and there's a lot of kind of similarity between these two which are kind of the interesting ones in my mind there's three the first of these is Apache beam and that's because a it comes out of Google which means that there's going to be a huge amount of excitement and interest because Google has been traditionally kind of a closed shop and by the time that they released something to the rest of the world they've already moved on and they're building something else this is kind of the first time the Google is releasing something that they're actively soliciting involvement and contribution from the community so it's really kind of their first I would argue real foray into the open source world and the notion would beam is that it's not a streaming platform in its own right but it's a way for people to unify all the streaming platforms that they may want to use for the use case under a single API so if there are some things that apex does well and some things that storm does well and some things that flink does well you can literally use them interchangeably because you know that they're all beam compatible and once you get sufficient momentum with this kind of approach this becomes super powerful right it's essentially defining a standard for streaming and that's a really great idea the next bit is the patching apex and that's up here because in my mind of all the open-source frameworks that are out there it has the most robust architecture and it solves the problem of durability and availability in the best way and the last of it is is IBM fo now this year is not open source tech I lost my mic I apologize this is not open source technology and in fact it is extremely expensive technology but it is also pretty much the gold standard for what can you build with a streaming system this is the solution that IBM built for the NSA to process all audio and video from everything that the NSA does so it has a you know insane performance never drops data extremely scalable but it's also IBM you will pay for ever and ever and ever to be able to run on top of it and in the open source world we we believe in collaboration and not paying for things so apex kind of has an interesting origin as far as as the other technologies out here and that both both storm and flank kind of came out of research land and apex was really a product built for businesses that became an open source technology and what that means is that their focus was different from the ground up it was built for reliability and strict processing semantics rather than just focusing on performance or buzzwords right they needed to build a product that other people could use and that really shows now like the kind of the key kind of message I want to send here is that apex just works and it doesn't just work as far as building a toy application it works when you try to take that toy application and run it a thousand times bigger on a cluster and it still just works without kind of these unexpected surprises and as key strengths are for dynamism scalability and and the way in which it handles failures and I'll dive into all three of those its weaknesses and this is a big one is that it's not that easy to use it has a learning curve and because of its more complex architecture right it's closer to storm than to flink you really have to kind of manually put all the pieces together but at the same time then you're not dealing with this curse of your high level API the first bit is that of the three technologies we've looked at apex is the first one that allows you to dynamically reconfigure your dag in flight one of the really interesting points from from the Concord presentation earlier is that this is now another open source streaming platform that allows you to dynamically reconfigured apology I think to date the only other one I'm aware of as kafka streams and I'm happy to be wrong there these things move quickly and it's hard to keep up with all of them but you can both adjust the connections within your topology so how operators connect you can adjust the applications that you deploy without influencing other applications and you can adjust the number of physical partition processing your data apex integrates natively with yarn and so essentially everything is just nicely packaged with in yard this is very much kind of your standard architecture diagram think that I want to point out that really matters is how operators are located within nodes and within threads so here we've got two operators in the same thread we've got another operator by itself within the same container but in a different thread and then we've got some operators in a different container and now this isn't an arbitrary distinction in apex this is actually configurable thing you have the option to say hey I want these processes to be threadlocal to each other me and execute in the same thread or within the same container or on the same note you also have the ability to say I want these to be in different containers in different threads in different notes and what you're really doing there is you're making a trade-off between performance and durability if everything is in the same thread if you've got an error or a failure in one of your operators that's going to affect everything in that same thread but if these are in different containers are in different nodes then you're protected at the same time if you're within the same thread then you're not paying the cost of inter-process communication or networked agencies now this last bit right we talked about this notion of having three parallel pipelines all doing the same thing in apex there's a unique thing and that is this buffer formally it's called a buffer server but this buffer server acts as an in memory blocker between any two physically separated containers and what this buys us is that when you have this failure errors do not propagate its failure handling mechanism is very similar to Flinx with the exception that it has the buffer server in the loop so it only needs to roll back to the previous checkpoint and that's again a configurable thing and it's configurable on a per operator basis so for the first time we have a system in which we can build applications where you can make trade-offs between durability and performance as a function of configuration this paradigm lets you make applications that you can deploy without having to run multiple applications doing the same thing you run one application you can run that same application in different clusters if you want even more redundancy but keeping this entire system running turns out to be a lot easier and I already spoke to this but apex supports dynamic scaling of how much data its processing and it's this scaling can happen both manually through through user input ever on time or automatically through stat tractors that you defined within the framework so to that it's kind of the best but it's hard to use right you you can fine-tune your data flows and your creation of your data to optimize performance and optimize for durability they have a really good UI that lets you see kind of all the nitty-gritty of what's going on in your system in terms of memory usage CP usage I oh where you have backups etc it has built-in back pressure but there is no high level API yet that's something that's in progress there's no machine learning support like flink has with flink ml there is a learning curve the same one associated with storm more or less and it's built for business a stop bill for simplicity it's not just something that you can pick up and use right off the bat but it is something that you can build a product on top of and that turns out to be really useful so the last bit is that I've talked a lot about streaming but there's kind of more to it than that I haven't touched it all on this notion of state and state does matter in streaming systems because you're not always going to be only dealing with in flight data and kind of the key questions here or what if I need to persist this data across operators but retrieve it quickly and then do analytics and build machine learning models for example on top of this data to actually give me useful insight and why do we want state the answer is that these are complex problems that we're trying to solve say we want to look for anomalies in your spending if we're trying to build a fraud detection system we want to be able to track your spending over time and we want to be able to persist that data but we also want to be able to query it very quickly we want to be able to do statistical aggregates to understand whether we're seeing anomalies in this data we want to be able to train machine learning models so that we can close the loop and quickly deploy updated models that more rapidly reflect changes in the ecosystem or the environment that we're monitoring and we want to be able to support cross operator communication because all of these stream systems partition data but at some point you might want to unify this data and group it and make insights across a range of data and these are hard things right and there's not really a good solution out there that i'm aware of it in the open source realm it really brings us a kind of this idea of distributed in-memory databases and there's a number of them out there there's their skill adibi now there's geode Redis has been around for a while mmm sequel ignite hazel cast and dhts which exist in you know computer science world in general but not really implemented in a complete fashion for you know having this in distributed in memory store that lets you process data but lets you also respond to queries quickly and scalable and these databases are hard to build because when you're dealing with memory memories volatile if you have a failure that data is gone right you're you're not going to be able to just reread from disk once your machinery boots this means that you have to replicate and when you have to deal with replication then in a distributed environment you run into the cap theorem right and it's harder to guarantee consistency availability etc and memory is ultimately still a pricey thing and it's a limited resource which makes it hard to build systems you know that you can just kind of go to town on with the way that you can build and MapReduce clusters and run them on commodity hardware and massive scales right that was the whole idea we can run 10,000 note clusters for four pennies so with that that's all I've got if anyone has any questions I'm happy to dive into any and all these ideas more you so is a great talk and thank you so much one dimension you haven't mentioned that is the if you adopting one technology for example if I using batch for the using spark you know for best process in detail what you're learning all that stuff and all sudden kind of streaming says I decide to choose something else and then all of a sudden my top technology stack is you know so how do you solve that problem a great question the we all heard that basically how do you deal with the fact that sometimes you're ganna want to do both streaming and batch and Blynken and apex for example will let you do well let you simulate batching right because of the windowing mechanisms that they have you can like aggregate data over time but then you run into the notion of aggregating data in your memory right you're essentially at that point need to persist state you run into the challenge of how do you build an efficient efficient you know machine for holding your state that you can then do van oolitic saan that's kind of one answer the other answer is you you take the Google beam model right you or Apache beam and you you have multiple systems tied together but when you have multiple systems then you're you're dealing with more complex infrastructure right there's more room for problems some folks have argued that streaming is a strict superset of batch and you know you can if you have enough hardware to throw the problem that's kind of true but there's not sadly a truly good answer for that yet I think we're moving in the direction where it becomes easier but at the moment it's kind of very you skate specific you mission apex is on yarn what about may cells can it be also managed by May so sore has to go through yarn at the moment it is purely yarn based I think maces may happen in the future it's not on the road map yet I'm with you I would like to see it on mesas as well but it would be nice to have that flexibility my stuff okay alright so the epic song I mean yarn so he's a yarn is a long-running young container in that case a good question nope so in apex application each operator uh will run configure ibly in a in a different container essentially right by default every operator will be its own in its own JVM so its own yarn container but because of the locality configuration that you can specify when you're creating your topology some of these operators will be condensed within for example a single container right so this this kind of lets you make that trade-off programmatically between like in storm you kind of have no control over what happens so if you want things to run independently so that you can debug easier or so the failures don't cascade between you know different tasks you can't do that in apex and in flink you have some level of control over that there is a start-up costs associated with yarn there is a startup collar there is a start-up costs associated with recovering containers when they fail in apex so FX is a self-healing system anything that fails including the application master and the app manager will recover it does take time now the one thing that in apex we have then we don't have anywhere else is that when you do have a failure it doesn't affect the rest of the system these failures do not cascade in terms of either you know other operators shutting down or in terms of the day actual data processing itself the one exception to this is that if the app master the the actual process it communicates with yarn and and does the research negotiation goes down in the current version of yarn that they run with all the operators in the system will shut down so that's kind of the one caveat the new it but that's a yarn problem that's not a a pex problem in newer versions of yarn you can you can get around that but it doesn't currently support last question ok Jane looks like thank you ran out of questions before the last question