Introduction to Transactions in Apache Pulsar

An Introduction to Transactions in Apache Pulsar by Jowanza Joseph

Recording: An Introduction to Transactions in Apache Pulsar by Jowanza Joseph

uh welcome everybody uh my name is alexis kravruf and i'm the founder and organizer of scala and now scala bay and vancouver scholar in addition so this is a joint meetup and this is the second talk of our last meetup of the year happy holidays uh we're very happy to have jawanda joseph with us initially he planned to give this talk at scale by the bay 2021 it didn't happen but we are very happy that we can actually hold jawanza here and we have more time and obviously uh you know you guys are welcome to ask questions through the chat and then we'll have a q a at the end um jawanza introduce yourself and i'm thinking the topic is super interesting so welcome and take it away yeah uh let me share my screen real quick and tee up this presentation um okay yes so um yeah i am jawanza joseph i live in the salt lake city metro area in in utah and um yeah i've just finished publishing a book with o'reilly about pulsar and so i decided to use this opportunity to talk a little bit about pulsar and then also know the pulsar um client the scala client so um i'm gonna just i think this should be around 40 40 minutes or so that's kind of what what i had planned and so we'll go through a little agenda i'll talk a little bit more about you know my background in a second and we'll we'll go from there so the agenda we're going to talk a little bit about exactly once processing so what that means in in the context of message uh processing uh what delivery guarantees mean what trade-offs you take when you you enable those different you know guarantees and then what uh the approach that pulsar takes to the specific um tract and then we're going to talk about transactions so i'll provide a little motivating example for where we want transactions in you know messaging system and then we'll talk very detailed about pulsar's implementation of transactions and then i will provide a two actually end-to-end examples of using transactions in pulsar with pulsar 4s which is a it's basically a wrapper around the java library that has kind of some more idiomatic scala you know api so one thing i will say so i initially you know had my my pulsar cluster up and i was gonna actually run run those examples but um i had some issues with my vpn and whatnot so i'm just gonna walk through the code rather than rather than embarrass myself and have things not work so um just a heads up on that and we talked a little bit about myself uh just as an introduction it's my first time speaking in this group so i work at phoenicity which is a open banking company that's owned by mastercard and i started there as an individual contributor and i recently got promoted to vice president so that's been weird and i'm still still getting used to it uh so i am also the author of you know mastering apache pulsar and i've worked on this distributed pub sub stuff most of my career so um it's been it's been pretty fruitful and yeah i do road bike as well i've got this picture of marshall taylor uh behind me who's a noted um cyclist and uh i i just love to love the sport so i yeah i tweet about it a bit sometimes okay so let me just do a quick intro to apache pulsar so what pulsar is is a it's really you know more than a messaging system or a distributed messaging system but it's it's intended to be really a new paradigm of how to store streams of data so it's really built you know from the the ground up to scale the storage and retrieval of streams and so pulsar is kind of composed of three pieces mostly well but there's more pieces to the ecosystem but the core parts of pulsar are pulsar nodes which are jvm your processes that that run and communicate with pulsar clients uh zookeeper which is an apache project that is used for both kind of configuration management and storing of ledgers that um your pulsar nodes utilize and then apache bookkeeper which is the storage engine for pulsar so one of the key differences between you know pulsar and something like kafka is that the storage is is separated you know from the you know consumption and um and the publishing and the consuming of messages so the the storage is is all handled you know in bookkeeper and then pulsar nodes are really stateless and they're they're just there to kind of coordinate between the clients and bookkeeper like deciding kind of how to handle uh you know acknowledgements and things of that nature and zookeeper is there to kind of keep the distributed kind of coordinated state and so pulsar um it's it started as a project within yahoo and then it was open sourced several years ago and it's been you know growing in an adoption and there's a lot of interesting things about it that are outside of what we'll talk about today we're going to try and focus only on on transactions but if you're if you have an opportunity to try it out that i definitely recommend it it's pretty easy to get started with um so let's talk a little bit about um messaging guarantees so we're going to talk about at most once at least once and exactly once it looks like my subtext here got kind of messed up still has the default text so we'll just move past the slide so at most processing is um illustrated by you know this this picture here so we have um you know two iphones we are sending a message from one iphone to another and we expect that the other iphone will receive it but sometimes it doesn't and that at most once processing uh we will just kind of live with that outcome so we will not you know try to try to send another message to append you know this this situation we'll deal with the idea that you know we might not you know get all the messages right we we don't want to we don't want to increase the complexity of dealing with with getting multiple messages so we just accept the idea that there's some you know percentage of messages that could be lost in the event of your different types of failures whether that be a broker failure or client failure we are just going to not not handle that and many uh messaging systems uh have this semantic because it's fairly easy to deal with and to kind of reason about as a as an end user but it's you know has some some trade-offs to using it so you know the pros you know would be that it's pretty simple so to manage this you know type of uh semantic you know there's no duplicated works you don't have to think about as a you know consumer client on the receiving end like how to kind of manage multiple messages that are the same and it's oriented toward performance so you you can just you know send send and forget as a producer and not think about you know trying to trying to resend or even from the broker's perspective you know not trying to keep track of everything that's been sent but the limitations you can't have your data loss and end-to-end system and you can't necessarily use it as a system of record because there might be you know some message missing messages and it can create some mistrust so um if you've ever worked with your product teams and you have a system that's at most wants in a message delivery and you know something goes missing and somewhere in the pipeline they you know kind of freak out about the you know the size of the issue and so it does have those kind of you know both technical downsides and then you know the kind of human oriented downside so at least once would be this type of interaction where you want to make sure every message is delivered and sometimes that requires you know re-requesting the message again so uh you'll end up in the scenario where you know as a um you know the sender you might have to send the message twice if the receiver you know doesn't receive it so in in the terms of in in terms of messaging systems it doesn't exactly happen this way typically the broker would be you know the one that would you know kind of kind of manage that and rather than you know the you know the end phone here sending it twice but the the idea is the same right where we want to just make sure that the receiver here john you know gets gets the hello message and so if if that means that we are unsure that he got the message the first time we'll send it again and if he gets two then we're willing to kind of deal with the the consequence of that which you know is kind of variable on what those consequences are so um the pros are you kind of guarantee that all messages will be delivered you know to the consumers which is a very good pro the cons are you know it does require the duplication on the consumer side uh so the receiver needs to realize that you know some of those messages might be the same and they're not new not new information and they need to implement you know some solution whether that be around identification or um you know the working with the broker and tandem to do that which is is more work and so it does cause some performance issues at scale and then also can be an orchestration problem as well uh where if you have you know more of a topology or a like a dag you know of messages being passed you know there you have to kind of think about that you know message duplication problem all throughout uh but you know similarly to the at most ones you know this at least once is is pretty common you know in many messaging systems to be either enabled or sorry the default behavior um in those because it's you know for many applications this is a fine outcome so if you think about something like sending sending an email for example um you know about a sale like you you prefer to you know send two and maybe annoy your uh receiving party then then the possibility of not none being sent and the um you know the value or the degradation and opinion of receiving two emails is you may be not as bad as receiving none and missing the sale um and having kind of you know extra load on your maybe your um customer service team so that's a trade-off that you you can make in these types of systems so exactly once uh would be that every message you send from mom gets to john so mom doesn't have to you know think about or worry about your messages being gone and john can worry about um you're just receiving messages from mom and not you're thinking about any any kind of alternative scenario happening um as you can imagine right with something like a cell phone network you know having exactly once would be you know relatively non-trivial to sort of implement and then it's the same thing goes for something like you know pulsar or or kafka like trying to implement this you know requires it has have a kind of a large set of requirements in order to to get it right and we'll we'll talk a little bit about that in a second uh so for this you know you get the guarantee of you know all messages are processed you don't have to think about duplication as well but you know the implementation to do it is much more complex and typically requires you know all three parties that are involved so the consumer the producer and the intermediary broker kind of all need to be kind of orchestrating it to really make something like exactly once work so what does exactly once look like in apache pulsar so apache pulsar has this uh concept of message deduplication um and so the deduplication is kind of the basic way to you know ensure um you know effectively once is what what the documentation says are not exactly once uh semantics without um you know introducing a lot of performance bottlenecks and too much complexity so uh let me back up a second so in a typical uh producer scenario uh the producer you know will will send a message to the broker and the broker will kind of acknowledge the receipt of that message and um in with message deduplication you add an extra essentially like a key or or a sequence uh to your your send to say this message was you know number one and then you could say this next message was number two and so what this does is within the broker it gives it a and kind of a effective ordering right of of all the messages within a topic and uh if you look down at the bottom where the producer you know sends you know send one again with the message what happens in that scenario is the broker will acknowledge it but internally you know it knows that okay i've already received you know one uh in in this topic already so i'm going to ignore it effectively so this does have the advantage of like you're providing a way for the producer to really kind of control and know like every message that's being sent to the broker and the broker a way to you know keep going in the normal flow of send and not acknowledge uh pattern but it gives you this extra kind of layer of protection around you know receiving a duplicate message without you know breaking anything considerably and then from the consumer side um you would also you know receive the your message with with that that sequence and so you could utilize that sequence as a consumer to effectively be kind of your offset in the in the subscription or topic or you can just use it to be a kind of a reset point so if your consumer were to go down you can store kind of your place in the sequence and then you know replay from from that point so the you know pros of this are it's really high performance right it doesn't really change anything materially from you know a normal you know publish subscribe pattern and pulsar and then you do get this effectively once which you know kind of asymptotes to and exactly once semantic uh however you know there is some complexity to think about so in in the scenario we talked about you know we we effectively just talked about you know one partition so a topic that has you know one you know producer one consumer one partition keeping that sequence straight um is it's pretty simple but if you were to get into a multiple you know partition topic then um it's a little bit more complicated because now you know the the sequence isn't in order you know per um like per topic it could it could be um you're different and so you have to kind of think of a strategy of how you will ensure you know that same kind of sequencing or how you would handle that which is it's complicated um for sure and then it does require you know producer side and subscriber side logic for it to really work so um you know like we we talked about earlier um that's kind of one of the downsides of this is that it's not like you you just send messages to the broker and you know it just kind of automatically produces this semantic it's really um it's really everything working in chorus right and every component of it has to really kind of be dialed in for it to work otherwise it it just doesn't if you have any any kind of mismatch between the producer and consumer or broker then kind of all guarantees go out the window but if you kind of really tie together the sequence id the message you're all the way through then then it will work so it's it works it's performance but you know there's some it leaves a little bit to be desired around your multi-partition topics and then you know more maybe complex use cases around those so um what's a transaction then so transaction and pulsar you know allows effectively two things so it allows a producer to send a batch of messages to multiple topics where all the messages in the batch are eventually visible to any consumer or none are visible ever or none are ever visible to consumers so that's kind of a mouthful but effectively it works similar to what you would see in like a database transaction so you know you start with a database transaction you open it and then you try to you know do a number of operations and then you commit them at the end or you can abort them so it's the same concept here where you have your consumers and producers you have topics and you want to you know produce you know multiple messages like either you're across many topics and then you only want that to happen if all those things happen in sequence and if they don't then you want to roll it back so that it doesn't leave the kind of overall system in a inconsistent state is is the idea behind transactions and it also you know provides uh an end and exactly one semantic so uh we'll talk about this in a second but pulsar you know has this idea of consume process and produce so uh you know you're a producer or sorry a consumer you can see messages and then you do some action to it um and then you produce it to a new topic or or multiple topics or a partition topic and you want that all to happen you know in in a transaction so that this provides a way for that to happen and we'll we'll talk a little bit more specifically about how that works it's it's actually fairly comp but it's i think i'm maybe a little bit um a little softened to the complexity of it because i just spent like my year trying to write this book but it it has some complexity to it but it hopefully will kind of elucid but like illuminate like what what's required to make this work just talk about a little bit about a motivating example uh so i joined this gym once upon a time where um they paid you to lose weight so you effectively sign up and you say hey i'm going to lose you know x amount of pounds and x amount of date a y amount of days and then they would basically loan you that money and say okay you can use this money in order to kick start the process of losing weight and if you don't do it then you need to return the money to us and actually just return it with interest um so it's supposed to be a motivator where you know every every time i would weigh in have some kind of goal then if i didn't make that goal then i would need to essentially pay pay the gym um and so and then if i didn't want to be in that kind of arrangement anymore then i needed to like end my contract and there's a fee to do that as well so uh the the reason why i think this is a reasonable motivating example is um you know i i can make this promise right that i'm gonna lose 50 pounds in 25 days the gym can give me that 100 bucks and then um you know at the end of that 25 days you know the base of the transaction is open you know that for that 25 days and then it closes and either you know i am 150 pounds or i need to return money with interest to to the gym so um you know that it could be modeled as you know a either like a database transaction you know or a message transaction um in pulsar where you know there's essentially like expiry on the initial you know contract and then you know after the 25 day period you know either it kind of unwinds itself or you know pushes itself forward so back to this semantic with with pulsar of uh you know consume your process and produce um i think just just drilling in a little bit more into this idea it's it's really about you know providing a framework to you know perform more kind of like complicate complex tasks and and really kind of push messaging you know to the next frame because a lot of the the thought and design you know around around messaging is uh this this concept of like you you have your your topic you know you have your consumers you have your producers and you know there may be a little bit of uh the complexity in the the in like the topology you know of your your topics but you try to keep things simple because kind of the more complexity you add and the more dependencies you add kind of the more you push yourself into a place where the guarantees maybe won't follow with the application you're trying to build and so having a way to be much more intentional about what you're trying to do saying hey these all need to occur you know in in one like transactional process is is important and it does enable you know a lot of new use cases you know in in message processing and so um it's i think this is a good framework for looking at it and um let's talk a little bit more about like what this looks like um it's it's going to be i would warn like introducing a number of like topics or you know concepts and pulsar in order to really grasp it but i think it will be worthwhile to to go through so let's start with the first thing that happens so um there's a lot of things on the screen and we're going to try to walk through each of them as we need them so in this diagram where you see the blue like that's all we really need to talk about for now you know the the box that has the pulsar client the coordinator and the transaction log we're going to start with that and then you know as the transaction process continues then we'll introduce more concepts afterward but we'll we'll start with that and i will also say um this these diagrams are from the documentation for pulsar and their most recent uh release because i think that you know they're not their diagram is really the best you know i have one that's kind of approximating it but i think this this one is much better for like illustrating kind of how this works so how this works is that you're a pulsar client right and you want to you know make a transaction so you want to start kicking off the process to make a transaction so in your client library we you effectively say i want to make a transaction and what that does is this 1.1 step where it will essentially send a message to the broker which the coordinator is a process that runs on the pulsar broker and it will say i want to make a new transaction so then the coordinator has something called the transaction log which is an immutable log of all of the transaction related events that happen for that coordinator uh so you'll say hey i need to make a new transaction you know for this pulsar client so then um at that point it will you know append that to the log and then it will send the result um back to the client and say all right here's a transaction id and you can use this transaction id now to go ahead and make transactions to do whatever operations that you need to do within that transaction um so that's that's step one of the process okay so step two things get a little bit more complicated so uh what happens here is that this pulsar client that had the transaction in the previous step decides that hey i want to produce two messages to two different topics as part of my transaction so the first thing that's going to happen is that it is going to do 2.1.1 where it will tell the coordinator hey i am producing you know this message um you know it's a partitioned topic with this transaction and it will add uh those two events right so we're adding you know transaction uh so transaction one's id excuse me and we're adding two events essentially we're gonna do in produce from one and and produce from two and so we're gonna do that and it's gonna you know send that result um you know back to the the initial producer and it will do the same you know for producer too but the producer will send you know this message to the broker and then those brokers will write to the log uh that there's you know a message for you know this topic uh for producer one and this topic for producer two uh so at this point you know there's not a nothing is committed so there's not a um this is not like a fully kind of end to end thing that happened this is all sort of in process right we're still we're starting the transaction we're doing we need to do in order to kind of bring the transaction to fruition but as a consumer of either of these output topics right you wouldn't see these messages quite yet because they're effectively in a buffered state all right so they're in a state where they're going to be you know written to the topic as long as the rest of the transaction components complete so go into step three um so step three would be that now that those two messages are produced you know so tx1 m1 here tx1 m2 we will want to consume or acknowledge you know the receipt of that uh message from the other side of the the transaction so we're um you know effectively producing and then acknowledge that we're receiving that so first thing we need to do is send back to the coordinator again and say hey i am you know subscribing to you know this topic and then also specifically to the subscription um and so then that's so then we follow that same path of like you know logging you send in the results back to the client and then from the broker side uh we're going to go ahead and do the same thing that we did on the producer side where we create a you know pending log of like this is a pending acknowledged you know transaction so these two messages were produced and then they were consumed but we're pending until this this is all committed uh so so far we've you know started it we made we asked for a transaction uh we produced two messages we acknowledged it and so we so all of this is transpired we're still in a pending you know state nothing is you know committed to the real you know kind of topics to to be consumed by other new consumers we're still in a you know pending state that nothing is visible you know outside of the the coordinator and then these sort of like you know temporary um immutable logs for for the transaction uh the next step would be now the pulsar client that started the transaction uh needs to kind of like end or commit the transaction so that will be you know a command that would go to the coordinator again and that would be added to the log that's being committed uh and so that's the the fourth step to kind of like get this from a space from where it was all pending so now like okay we're happy with how this turned out so let's end the transaction and then uh the next step would be this which is it's kind of complicated but let's let's walk through it slowly so the thing that needs to happen now that the client has said that they want to commit is that the coordinator you know could be you know one of any brokers and then the brokers that received the you know acknowledgement and the produce uh messages it could all be different brokers so now it needs to all sort of reconcile you know what what's happened or transpired so the coordinator would effectively knows from this log like which brokers are part of the transaction and so it will utilize that information to then go back to the brokers and and effectively ensure the state of the transaction is in the correct place right so there's a you know acknowledgement you know there's a committed you know um transaction for the broker and then that there's you know pending messages in in the log from the producer side so um there's there's a few little green dots that are like right right marker um on here don't worry about those for this this case but effectively it's an action to go and reconcile the state you know across all the brokers who are involved in making the transaction and then uh once that is confirmed uh what happens is then the coordinator will write a final kind of committed message to its log and then send a result back to the pulsar client saying all right you're um your transaction's done and then you know this uh pending acknowledged data log will move you know to an acknowledged message in this input topic uh so the semantics are are pretty simple we'll walk through it from the the um scala or sorry the pulsar for scala library but like the behind the scenes here excuse me it's pretty non-trivial right there's a lot of acknowledgements there's a lot of logs you know there's a lot of um coordination right there's even a thing called a coordinator so in order to make this possible um and then there is you know transaction id that's kind of being passed around everywhere and stored in all these different logs and then there is a um you know there's a necessity for each of these actions that happen from this client like consuming and producing you know all utilize the same transaction id to just ensure you know this kind of end-to-end flow works so the i guess the answer is um like why right why why are we why are we doing this and i guess back to the initial point is that you without this if you wanted to kind of do the same flow right where you you had a client you wanted to you know start a transaction you know produce messages to multiple partition topics acknowledge them on the other end and then close that transaction uh it would be hard right to to do and to guarantee um that there's no either data loss in that case or there's no data duplication i'm in that case whereas here you know everything is sort of really finely acknowledged and really finely logged and so there's there's not that kind of possibility to introduce you know that type of failure scenario so in order to do your transactions in pulsar it's actually a pretty new feature so the most recent version of pulsar is 2.8.1 um and 2.8.0 uh is where where this is kind of like widely introduced and then you have to enable you know transactions on the broker level there's a couple settings that that you need to turn on because by default it's kind of off um and then also the transaction has to be enabled on the client level as well uh so you know there's a little bit of you know overhead to manage to do this but you know compared to the the sequence and id for the deduplication um it's i would say kind of a smaller lift and also a different use case so it may be not really not worth comparing all right um so let's get into these these two examples um you know for uh scala four four s uh sorry pulsar for s i keep saying scholar for us it's been been a little bit of a long day uh so the with the two examples we'll walk through is one is a an example just using the async client in um your pulse r4s and another one that's using the uh i think it's the zed io client and so it's these are pretty simple examples that kind of follow the same um the same outline to like walking through this you know transaction now you can you can do much more like complicated things with transactions but i think for the purposes of you know this this conversation i think that it's a good good place to start so let's go to that um okay so um this is a transaction example um there's a few imports you know from this pulsar for s library um and you know it's a very very simple simple example so let's start you know on line eight so uh we've just are using a schema so pulsar supports um several types of schemas so supports uh you know different um primitive type schemas like like strings or booleans but then it also supports you know pro buff avro json and then with the pulsar 4s client you can you know set up some like more you know complex you know types of uh clients where you can do things like like tls or you know token based authentication but for this example it's just using is as if you know there was your pulse service installed on my local machine on port 6650 and i'm just turning transactions on here in line 14 for that client so um effectively what i'm going to do is just you know create a transaction and then you know create a consumer to receive uh the transaction or to receive a message and then kind of close the transaction and commit it at the end um so the so line 17 is me setting up my producer to read from that uh this topic that's online 10. line 18 is me setting up the consumer to do the same thing and to subscribe to the topic uh with a specific subscription name um and then we essentially you know tell the consumer to look to like the earliest offset in that topic uh which in this this case since it's a topic that that's just being used as like an example will be kind of empty so um then we have this you know comprehension where we first you know tell the client that we want to start a transaction and then we would have the consumer you receive you know an async message uh then we would wrap the produce threat to transaction in a message that that we would send um and then we would you know wait for that acknowledgement and then commit the transaction um and so it's for this type of you know interaction where we're just trying to do you know one or more kind of producing or we're consuming um it's pretty pretty simple it's just kind of a matter of you know wrapping your consumer and producer um you know wrapping the producer and consumer either acknowledgements or their sins and and so that would effectively create a you know transaction you know create a consumer that's waiting for it and then kind of transact around all that behavior um one thing to note is that this example it doesn't include like a boarding so so instead of you know committing you could also abort and then that would effectively like unwind you know all of this um from happening as well so um that's that's all really there is to it um from like the the simplest use case here and then um your pulsar 4s also supports like a few know different types of um popular you know scala io based libraries and so um the other example here is just with you know you know zio so um all of this you know stuff from five line five like three line 18 are all the same uh the only only difference is now uh we're adding an additional constraint so one of the things i didn't mention is with transactions you can you know open them with a specific time frame in mind so you're kind of putting a time bound on how long you expect the transaction to take which is a good idea because otherwise it could kind of be open you know indefinitely or to whatever the your default time frame is for for those which could be you know too long for your use case so in this case it's got a specific like one one second um expectation for the the turnaround time on the transaction and then um it's the same you know behavior here where where you have a consumer that's ready to receive this message async and then you know producer that's wrapping their product their message in the transaction and then it's being acknowledged and then you know yielded and then you can kind of close you know the consumer and producer after that so um yeah effectively this is like kind of all like the basics of the api right so if you wanted to use this to build something much more complex like you could but yeah most of the the use cases are kind of like narrow to hey i have you know this um event that i want to make sure it gets produced to these like 10 topics and if it doesn't get produced all 10 like i want to unwind it from all 10 um and so that's the the idea there so the pros of this um whole you know rigmarole that we talked about is you do get exactly once you know processing and then also you can batch these so if you wanted to do like much larger um you know interactions around you know messaging across multiple topics you can uh it is more complex right so you do introduce um you know a lot of new concepts like a new api and then you know there's a lot going on behind the scenes to ensure that your transaction works and um also so there are some performance implications uh here as well uh they're not really well documented uh you know from the maintainers of a pulsar so so far but there's a bit out there about about that and there's some you know work right now going on to really kind of catalog it and make sure that your users of it know and then right now it is limited limited to java api so fortunately scala you know oh sorry pulsar 4s you know is a wrapper around the java librarian so um some folks were able to kind of like quickly add this functionality to the library but if the pulsar does support you know go and um python as well and uh those clients don't have uh you know the ability to do this um and then like pulsar functions which is a like a stream processing runtime for pulsar doesn't support this um yet pulsar io which is you know kind of a kind of a plug-and-play framework for doing um like change data capture and other types of uh workflows and pulsar that doesn't support this yet either it is on the road map but it is a limitation so if you you know want to extend this functionality beyond you know just pulsar you know clients for you know these consume and publish events like it's just not not there yet so it's pretty early um and so summarizing like transactions are pretty valuable um and like they they provide kind of a new new way to think about you know what's possible in messaging and you can achieve exactly one semantics and pulsar you know one through the the duplication but then also through you know this transaction mechanism as well and that's it for me and happy to take any questions thank you this was awesome and the gold was very clear and beautiful uh thanks for sharing this uh any questions um i'll ask the question so uh i i'm assuming this was driven by your application so what kind of was the most interesting finding you found when you deployed pulsar obviously you know there's a lot of questions between pulsar and kafka what's what makes it uh interesting for you and obviously you guys invested in this uh api yeah yeah so i think um a couple of questions in there but like the i think some of the hard things about deploying pulsar is um you know going back kind of maybe to the the first slide um here is an early slide it's a lot of stuff right like you know you you like standing up zookeeper and bookkeeper is sort of it's not non-trivial you know there's a lot of work that goes into it and um that's maybe one of the downsides of it right so um there's a couple companies that are you kind of becoming the uh i hate to say it but the the confluent you know pulsar to kind of you know bring kind of ease of use to this and then like you know make it kind of easier to focus on developing applications which i think is it's good and um and i think that is the way i recommend most people go like this it's just too much to manage but as far as like using transactions um i would say that the the semantic to me you know seems very simple but to my team excuse me to them it seems very complicated right they don't really unders they had a really hard time understanding like what the use case was and then how to leverage the use case because um i think that for a long time we kind of really encouraged simplicity and like the idea that you would try to like publish to you know multiple topics all as part of the transaction or you know consume process consume again process all as one transaction seemed kind of like insane and so it's taken a while to really get to more of a hey this is these are the kinds of things we can do to like leverage you know this this technology versus like our kind of fear of um complexity where we're just trying to simplify everything we can look at this as a way to make more assurances around our stream processing rather than you know just something that seems very complicated so hopefully that answer the question okay yes thank you very much any other questions uh you mentioned the book uh is this out already can you folks buy it um yes i think you can pre-order now um however i am supposed to get some free copies um i just have to respond to an email that i haven't in a week so if you are interested in a free copy you can drop me an email and i'll make sure you get one i'm just going to drop it a chat excuse me i've been recovering from a cold so excuse me but yeah you can shoot me an email i'll make sure you get a free copy um and yeah it's it's supposed to be january that it's out um so it's like i submitted the the final thing and then there's gonna be a digital copy first and then the print copy i think i get in january so that's um yeah i would say um i have a lot of opinions about writing books but definitely get a co-author never write a book by yourself it's a terrible idea that's what i did and um it's i think if i had half the work the quality would have been twice as good so like that the quality is is really good but i think i could have done even better you know if i could have focused on only half the book and had someone else so if you do want to write a book get a co-author that's great advice and congratulations it's it's a long process and you know we love posting book parties so we consider it you know holiday party and your book launch party and a streaming party as well because michael and you are sharing this awesome streaming context so thank you javanza thank you michael thank you everyone for this uh great meet up uh and you know have a great holiday season uh and we'll see you all guys and then you're here yeah thank you for having me and have a good one thanks