Scale By The Bay 2020: Keynote: Karthik Ramasy, Apache Pulsar @ Splunk

Scale By The Bay 2020: Keynote: Karthik Ramasy, Apache Pulsar @ Splunk

Recording: Scale By The Bay 2020: Keynote: Karthik Ramasy, Apache Pulsar @ Splunk

[Music] okay so the title of my talk will be uh how we adopted apache pulsar after a long evaluation on multiple dimensions so i will take you through the journey of how we evaluate the evaluation on multiple angles before we adopted it uh so this is me i've been the same director of engineering at splunk after our extremely activation before that salaxie pointed out i have been involved in streaming system for quite a while uh in terms of developing heron and developing pulsar [Music] so these are probably legal looking statements regarding splunk so so the title of my talk will be as follows uh first i'll give an intro to splunk then based on whatever the applications and other things that we see i will outline some of the streaming system requirements and we'll go through how also satisfies those requirements and then i will outline a couple of uh slides on how we use pulsar splunk and i will take questions so just to give an idea about what splunk is splunk is what we call as a data to everything platform uh where we are primarily focused on ide security devops and business process where we collect all the data and process it and and get real time in and get the insights that is used for security compliance and other it is several use cases that how you engage with data and we have a powerful splunk application that can drive some nifty dashboards so so when we are at splunk we saw a lot of emerging use cases uh that includes several applications one is real time monitoring and notifications and the second one is interactive applications where you get the log from interactive applications deserve decide what feature to keep it on and what witcher to turn it off so based on certain regions and other things then log processing and analytics more in real time uh then iot data analytics where the data is coming through certain manufacturing floor or factory floor and um but you want to get closer to the source rather than getting the data first into a system then post processing instead you wanted to get the data inside in the stream itself and then uh from a broader perspective there are a bunch of etl i mean the data that is then produced by several migration of devices and sensors needs to be cleaned up before uh gets into the appropriate insight mechanism then gets into the data addressed so though as a part of that streaming etl is a big use case and of course we want to have real-time insights or visibility as the data is coming through so that is a whole slew of analytics that you can do on the fly and as the data is generated and streamed through we might want to route the data into multiple different destinations depending upon the value that it provides whether it's a high value or low value depending upon those things we might want to draw to different destinations and finally there are applications which it requires some kind of even driven flows like in the case of microservices where one event is subjected to several steps in the processing before we reaches it uh requires a response so in those kind of situations we need a messaging and streaming systems so there are several dimensions that we evaluated as a streaming system that includes scalability how fast he can scale and how much how soon we can scale it down uh durability how water does the different type of durabilities that we based on the application requirements and how fault tolerant the system is and how available the system is then whether when we have multiple multiple teams and multiple tenants customers are sharing the resources how to isolate them when needed and what type of client language is that to interact with and what type of messaging models like streaming versus queuing and what type of persistence it provides then um how to ensure type safety in the in the clients and everything that you write into the system can be enforced and how easy to deploy into kubernetes and how how to uh operate the system and what are the research recovery scenarios in the case of uh one zone failure as well as in the region uh the entire region failure what is the total cost of ownership and how do you do observe the system for operability reasons and from because of the open source nature of the components how is the ecosystem or what kind of adoption it has and what kind of community then what is the licensing model so there's a bunch of dimensions that we need to evaluate so let me go through each one of them as uh as soon as possible so the first is scalability and traffic can widely vary while the system in production at least in splunk and the system needs to scale up with no effect to publish or consume throughput on latency uh then we need to support for a linear increase and decrease in current throughput as new um company or new nodes are added and when the system is stressed out as the new missions are added and the load should be spread out automatically rather than doing manual intervention at any given time and the scalability should be across multiple dimensions both as serving and storage so this is possible in pulsar because of the fact of the multi-layered architecture uh as we see there are three layers of architecture one is the compute layer which consists of the workers layer where it manipulates the data within pulsar the second layer is the messaging layer where you publish data into the messaging layer and consume data out of the messaging layer and that is where the broker layer is that can be scaled separately similarly the storage layer which provides a persistent for the serving layer when the data is spilled over to disk so that is a wookie layer and that bookie layer will be able to store the data and the tube data as an assignment needed and that can be scaled depending upon whether disc is full or uh or the disk or scale down depending upon layers so scaling down because of storage needs so so actually it's independent layers for processing serving and storage then messaging and processing are built on top of pulsar whereas the storage is built on top of apache bookkeeper and this all comes in a single package for apache pulsar so durability uh splunk has application with the different type of durability requirements one is persistent durability so in the case of in this case no data loss is accepted even in the presence of node failures or even entire cluster failure and this is useful for applications like security and compliance where we don't want to lose any data that captures one intrusion or a blip in the data a needle in the data but as a replicated durability it says no data loss in the presence of limited node failures because with by just viewing by replication if you have a three note replication you can survive up to two note failures right and these kind of replicated durability is good for collecting machine logs and processing them on the fly and transient durability so in the sense like uh yeah i don't need durability at all as long as system is up uh i'm fine with the durability guarantees so data laws in the presence of failure especially in the case of metrics data and which are just showing more of a trends so even if you lose the trend that trend will reoccur the reproducible because of the fact that the cpu usage has increased that trend will continue to be there so in this case some data loss is acceptable so pulsar based in its architecture provides different type of durability so how does it provide so when the data is ingested from a producer and it goes into broker then from broker it goes into the bookie and when the data is received on the bookie uh the data is f-synced so i am synced and flushed into the disk before the knowledge is generated from the bookie so from that point of view it gives you a persistent durability um so now brokers any given time or can replicate the data across bookies so in the in other words whenever you send a message that is given by producer to the broker broker can simultaneously replicate the data into bookies so then all the bookies will do the head sync and give the acknowledgement once you collect all the acknowledgement only then you can collect your acknowledgement back to the producer so by the time the data gets into the producer by the time acknowledgment gets into the producer you not only have sync the data but you also replicated the data because of the uh fsing and the replication we can satisfy both persistent and duplicated durability now if you wanted to turn on air sync for replicated durability you can do that as well so then the third wooden one where transient durability so that does not involve the bookies instead the data comes from the producer goes to the broker and broker acknowledges and it cashes in the broker now when the broker goes down you incur a data loss but that's only for transient purposes so basically three different forms of durability durabilities available so the third one the ability of the system to handle fault tolerance under different component failures um ideally it should be without any manual intention to certain degrees so pulsar can tolerate the broker failures as well as booking failures when the broker failures occur the topics that are served from the broker is shifted into the other broker and the system continues to serve data not only from a published point of view but also from consume point of view no copying of data nothing of that sort so which means brokers are essentially stateless and you can scale up and scale down brokers you can tolerate failures of brokers similarly like when you have a bookie failure the the system rights will shift it to the new set of nodes so for example in this case when the broker to fail the strides will shift to the other one three and four broker and the background process copies the data so that he can keep the replica factor up actually so now because of the fact that the pulsar is divides a particular data into smaller segment or particular partition in the smaller segment um the at any given time the data will be writing on any three nodes assuming you pick a factor of three uh if one of the node in the replicas goes down then the the current segment will be closed and new segment will open on a new set of bookies so so because of that little capabilities of the segment architecture and the right availability of the system is very high in the sense like as long as any three bookie nodes are available or three storage nodes are available for a replica factor of three you will continue to keep writing that data so without losing it so high availability so system should function uh continue to function in the cloud or on-prem and the following conditions if applicable when two nodes are instances filed or when available to zone or rack file now we saw some of the broker failures and bookie failures now let us look at the deployment of the cloud where you can have zone failures so bookies provide us some kind of rack awareness for using rack awareness concept we can uh bring up some nodes in zone a some nodes and zone b some nodes in zone c and when the broker sends the replicas to replicate to the bookies it will send to one bookie on each zone so that way what happens is when the segment when the zone particular zone goes down not only does the segments become unaccessible on that sony nodes um the reads and writes will shift it into the other zone b and zone c so that way like we can survive zonal failures very easily in the presence of zone and rack various data is available in other zones sharing and isolation systems should have the capabilities to share many applications on the same cluster from cost and manageability purposes and isolate different applications on their own machines in the same cluster when needed so this comes into the concept of multi-tenancy which is input into pulsar where each cluster has the notion of several tenants and each tenant can have the notion of name spaces and within each name spaces you can have multiple topics so those in addition to that you can control storage quarters at a particular tenant level how much uh flow control back pressure rate limiting those kind of um features are already available in pulsar that we can control soft only we can isolate the particular tendency based on software and if need comes we can move the tenant to a particular set of nodes or hardware elements so that where you can constrain some tenants on a subset of brokers and the keys so these are all the quota enforcements so requirement number six which is the client languages in this case how many languages we support we currently we support like five we added the node.js support which are officially supported by the project in addition to that uh there are a few languages dotnet uh then scala then trust support is provided by the community themselves but officially we support six languages in the by the project itself open source project so then requirement seven multiple messaging models i mean the some applications require collect once and deliver one's capability which is like a example is process uh s3 file that is skewed into the messaging system and ingest into splunk index and receive data once and deliver it multiple times which recently like a multiple pipelines coming in and share the same data for different type of processing and for these kind of this thing we don't want to use multiple system from a cost and operations perspective uh and then we don't want to emulate one system on top of the other because then we have to write a lot of emulation code that requires maintenance so luckily pulsar provides both messaging models from a messaging point of view it provides exclusive subscription and failover subscription from a queueing perspective we provide shared subscription and key shared sub subscription and since it's providing native support for both the different models we don't need any extra code for infrastructure level that require maintenance so though says in the streaming model or the messaging model you have the exclusive subscription at any given point one consumer will be consuming the data and if any other consumer tries to connect then it will get an error and also the data will be delivered in order and by creating multiple subscription for a different type of uh different subscription for different consumer you can consume the same data multiple times so then failover in case if the consumer fails you want a standby consumer to consume the data yes you can do a failover subscription again the data will be served in order uh when consumer one dies consumer two will be delivered the data with where the consumer one left off based on the cumulative acknowledgement that we received on the subscription and subscription maintains the state everything within pulsar db the clients need not should not no need to maintain any kind of a stated uh pulsar tracks kind of everything so on the other hand in the queueing models uh the data is round robin to the consumers so in other words it's uh each of the data will be given to part of the data will be given to consumers each of the consumer consumer 1 gets m2 and m4 while consumer 2 gets the m1 and consumer 3 gets m0 and m3 and so this is for the cubing model so because of that uh this is this is again a variant of the [Music] shared subscription which allows you to do route by based on keys to a particular consumer so the key difference between the two is selective acknowledgment versus queueing for versus cumulative across event and the selective acknowledgement every message is acknowledged and kept track and that is we used for the queueing model because each consumer can be going at a different pace and on the other hand and the failover subscription and the exclusive subscription we do accumulative acknowledgements acknowledgement the last technology uh message will you make sure uh will is an indication that the client has subsumed or consumed the messages that has come before and uh the cubing side we can track millions and millions of acknowledgements that allows the system to scale so the persistence um offload the colder data to lower cost storage and it could be either manual or it could be automatic with a configurable threshold and it should be transparent to publishers and consumers allows for nearly infinite storage at low cost uh compliant and security so this allows this is needed especially for events where uh secure is kind of in this type of application where you wanted to do even sourcing and where you want to replay the data from some period of time and look for some patterns so [Music] type safety splunk applications are varied one class requires fixed schema the other class requires fixed schema with evolution in mind and then the third class requires flexibility for no scheme or handle at the application level so we don't want to bring internal system for schema management uh and support for different types of schema type itself so pulsar provides a inbuilt schema registry that provides type safety to applications and its server side system enforces type safety and ensures that producers and consumers remain synced with respect to schema schema hd also allows clients to upload on a topic basis and schemas dictate what type of data types are recognized or as valid for the topic so requirement time is of deployment in kubernetes fibonacci is a popular orchestrated at the deployment environment and we should be able to easily deploy in kubernetes and the surface area that gets exposed outside kubernetes would be minimal and then segregation of some of the nodes receiving the data from external clients in a cloud-like environment should be possible as well and should be able to deploy cacd pipelines for testing and deployment as well so the key thing is uh we have a notion of a pulser proxy which allows you to have only one endpoint exposed on the pulsar side of things in front of an elb and that allows you to spray the topic traffic on the pulsar proxy which in turn directs to appropriate broker which is serving the topic so because of the major way for the pulsar proxy works it allows uh does not expose the kubernetes uh node add ip addresses and everything outside to outside world so but still the system functions like the way it should be and allowing to scale the brokers and bookies by using uh cube cube ctl scale replicas so it's as easy as to with one touch up a command we can scale up bookies brokers and proxies and similarly if there is a need the proxies which are exposed to the external nodes can be isolated into a separate system using a node labeling of cool kubernetes and isolate from the brokers and booking nodes which are highly critical right so that's what the segregated deployment means so then uh operability uh system should continue to online and continue to serve production traffic in several of the maintenance scenarios and some of the components will be self-adjusting so for example bookie should uh turn themselves into read only when ninety percent of the disk is full on the load manager allows you to balance traffic across the brokers depending upon how much cpu on the broker is being used and or how much memory is in the broker is being used uh similarly disaster recovery scenarios of course critical data uh from each enterprise flows through splunk products and customer exposed expect continuous availability in cloud or on-premise and required to handle data center failure seamlessly and pulsar provides both a synchronous replication and asynchronous application you can emulate a disaster recovery scenarios by having multiple pulsar clusters across different regions or across different data centers and um the producer produced in one of the data center and the data is asynchronous is replicated into the other cluster and the data is kept around there and now you can restart those producers and consumers when the other data center fails but there's a notion of a replicated subscription in which allows you to [Music] start the consumers from where they closely left off in the primary data center if you really want exactly once kind of uh scenarios for well then we encourage sync replication that is a stretch cluster um so that stretch cluster can provide exactly one semantics then the requirement 13 performance and total cost of ownership splunk applications are widely varied from earlier timing requirements sometimes the real time is a necessity which is 10 less than 10 millisecond and there are near real time which is less than few minutes and sometimes application requires high throughput in terms of ability to handle multipatter by day in a single cluster and we did a comparison with a detailed performance study with kafka and some of the performance experiments are and its results are as follows uh we these are the settings that we used we used a aws i3 8x large machines with 32 cp each and 244 gb of ram and four uh nvme disks and 10 gbps full duplex and um and a dedicated 7gbps to the uh ebs volumes and 20 large 8x large instances were used and one painted group for about 15 instances for running pulsar versus kafka depending upon the experiments that we're on and another tainted group is used for producing consuming traffic and we used some of the settings like message sizes of 1k each and the bad sizes of 128k and time for batching in the sense like uh one millisecond and publish it so this is uh we use the standard open source benchmark uh where the producers are produced into the messaging system and consumers are consumed and we the benchmark coordinator coordinates everything and captures the results uh so for we fix the throughput and we measure the latency and the resources required so in the case of one gigabyte in and one gigabytes out uh the pulsar ebs with journal which is with uh uh for persistent durability pulsar ebs with no journal means uh um it essentially like uh means without any uh sync fsync which means like we replicated durability and the pulsar nvme with the journal means the with the local disk not the ebs and the kafka is the ebs only because kafka does not provide fsing capabilities even if it provides it's more of a background fsync which does not provide the replicated durability that sorry the persistent durability that i talked about so as you can see the person in the pulsar is consistently lower varies between 5 to 140 x in terms of published latency and similarly in the case of one gigabyte in three gigabyte out again the latency is 5 250 x and again 32 gigabytes three gigabytes out again the variation in latency is pretty consistent across the board then uh essentially somebody one of the summaries pulsar provide consistency lower latency when you publish data and the number of brokers and rookies used for one gigabytes and one gigabytes out uh pulsar ebs use with journal requires a little bit of more 13 nodes that is because of the dual writes because we write in f sync into the journal as well as get the data into another disk on the same node because of the fact that uh data goes into a separate path unless journal goes to the separate path so which means nationally we are doing dual rights so that's why it guarantees uh persistent durability so that even even if the entire cluster is down you don't lose data at all so for that extra durability you pay some cost for it on the other hand with um [Music] if you're doing replicated durability our cost is much cheaper so essentially like uh 20 to 40 percent less brokers and bookies combination than kafka uh compared to the eps and the nvme on the kafka side of things with the additional 30 percent more brokers and bookies you can do persistent durability uh this again the same cases with respect to 1 gbps and the gbps out and the disk right bandwidth so pulsar essentially provides [Music] a lot of bandwidth et exploits a lot of available bandwidth per bm when it essentially writes into the disks so if we use nvme we can get up to 850 megabytes per second whereas if we use ebs because of the fact that ebs is provisioned based on number of iops we get we can get as much as 530 whereas when we with journal we do 250 because we've got the dual right because the ebs uh data path is saturated because that network bandwidth has to split between dual right so that's how you get after bandwidth on that but kafka is producing only 350 compared to pulsar again the same observations so pulsardine uses 20 to 30 percent less brokers on bookies since his exploits are available this bandwidth uh very efficiently and we have some more additional performance improvements coming as well so from a cpu usage point of view and the amount of codes that is consumed is much lower 40 to 60 less cores than kafka kafka uses more cpu due to crc overhead computation and scalar overhead that we saw again it's the same with respect to 3gbps out as well uh nicu says nick usage is more or less the same uh because for when you do multiple replicas on both system the system the publisher has to go into multiple gold via network not to publish to other brokers so which is um perfectly so pulsar uses 50 to 60 percent less cpu cores with complete control of memory and so then the number of vms needed so couples are uses 30 to 60 percent less vms than kafka this is due to the effective use of the bandwidth per vm by pulsar and that happens in both the cases of one gigabyte one gigabyte in one gigabyte out and one gigabyte and three gigabytes out and the number of mediums needed in pulsars as you can see is much lower compared to this so essentially the summary here is pulsar uses 25 to 50 percent less vms for the given throughput with additional 30 percent more vms also can support extra durability which is if we need it so single partition throughput one of the key thing is uh how much whenever a topic is divided into partitions how much throughput we can get out of a single partition so pulsar actually provides a much more higher throughput it's not limited by the disk bandwidth whereas because of the underlying architecture in kafka where each partition is slotted into a disk you are limited by the bandwidth of the disk whereas in pulsar you are not limited or not by the bandwidth of the disk instead you can have several disks based on striping and other things that you can do at the storage level you can even get a higher throughput so single partition latency as he said like the latency is consistently uh 5x to 100x in terms of even single partition as we saw earlier in the earlier experiments so the summary it's a pulsar is 1.5 x to 2x lower in capex cost with 5 to 50 x improvement in latency and 2 to 3x lower in opex due to the layered architecture so so this is again the summary of the performance whatever we did so the next requirement is um observability so when in production we need visibility into how the system works and you need to know the overall health of the system and the system should expose relevant detailed metrics i should be allowed to easy to troubleshoot so it also provides pre-packaged dashboards in terms of system overview metrics messaging matrix topic metrics broker metrics booking metrics proxy metrics and jvm log metrics the slew of metrics are available that allows you to troubleshoot the system very quickly so the ecosystem pulsar ecosystem is growing pretty fast in terms of the number of components and integrations are being added i came to know that a company that is the financial company that is going to adopt pulsar is going to do around 3000 integrations and adoption is uh over 600 plus companies and still it's growing at a faster pace and we are one of the we are proud to say we are one of the adopters and we also commit uh bug fixes and features and everything back into open source because it's in our interest to uh grow the projects so we are investing a lot in pulsar so community also is pretty good and so we have 320 contributors last that i saw and uh 6.7 gig cube github stars um we have around 600 plus companies that adopted the pulsar and we have 30 full-time committees contributing to the project from different companies and the cool thing in apache pulsar project is everything is apache license and it's not vendor control um so that allows for a lot of features that is available in the pulsar project can be consumed without the fear of the license being changed later and other things based on the because of its center control so that way like uh it's a pure apache license play-based project and so if you want more details about apache web also versus apache kafka here are some of those features so there's tons of them so how we use now let me come to the highlights of the talk how we use pulsar at splunk so pulsar is running as a service in the cloud for processing several billions of messages and consumed by several teams and pulsar is integrated into one of the flagship product called splunk dsp which is a stream processing offering from splunk [Music] that is a core stream product and from 1.1.0 onwards it is running with pulsar uh currently we have released 1.2 by end of october um and apache pulsar is being introduced in several other initiatives as well um which will we talk about it as the products become available splunk dsp what does splunk dsp do long dsp is a real-time processing solution that collects process and delivers data to splunk another destination in milliseconds in other words it allows you to collect data from my right of sources and filter them enhance them normalize them you can transform the data then aggregate format the data and mask sensitive data detect patterns or conditions and turn this raw data into high value uh information and protect any sensitive data and distribute data to splunk or other destinations depending upon how the data should be routed and so if you look at the dsp architecture we support multiple protocols for data ingestion the splunk's http even collector protocol and splunk to splunk protocol and any even the data sources could be batch oriented as well where the data is periodically uh collected and sent into a streaming fashion and once the data is collected the first thing that hits is the pulsar so pulsar is completely used by the um uh access the way to capture the data and then the stream processing engine picks up the data from a pulsar and process the data and sends it to the splunk index on any other external systems so pulsar is at the core now pulsar's footprint is increasing within dsp itself the all the connected framework and all the various things for batch and streaming connectors where the data is being written out from the dsp it's all going to be in pulsar so closing remarks splunk is committing to committed to advancing pulsar as it is used by our core products and cloud services so a lot of current work is going on and uh we have improved the good client in pulsar i mean based on some of the uh issues that you reported within the teams we used by the teams in splunk and we added the support for batch connectors and we added a pulsar uh kubernetes operator and we have got critical bug fixes done based on running production at a high scale and these some of them has been already propagated back into upstream into the open source and we are in the process of committing some of the stuff back into open source as well um and the feature work uh we are going to take up auto partitioning where topics need not be um associated with any partitions instead of the part partitions will be a hidden concept where you just create a topic depending upon the load it automatically adjusts itself then we are going to enhance the data store uh the metadata store so that you can plug in any metadata store that you want uh so that it can co-exist with other environments within an enterprise then we are going to enhance the state store which allows you to store state when you're processing uh in the functions and the all the pulser functions are pulsar connectors which is deep state in order to cut over from uh [Music] you know when you cut over because of failure from one function to another function the state could be retrieved so we have a booth for a demo of tsp if you are interested and we are hiring heavily if you are interested in working pulsar or in dsp please visit our booth and talk to some of our folks thank you [Music] you