Scale By The Bay 2018: Karthik Ramasamy, Creating a Data Fabric for IoT
Recording: Scale By The Bay 2018: Karthik Ramasamy, Creating a Data Fabric for IoT
Oh case the fabric for IOT with a project called Apache pulser how many of you have heard about Apache pulsar oh great I didn't expect to but it's nice to know so I'm the co-founder and CEO of a company called steam duo which was founded last year with a venture CDCA from Lightspeed and we worked a lot with Apache pulsar to provide an end-to-end messaging slash storage slash processing system so I'll be talking about how you can create an entire data fabric for I go to using pulsar so now let us since the time is short I will be running through the slides pretty fast because I have around 30 slides to cover so anyway so we are in a connected world where data is coming at a container large velocity and volume in a continuous fashion to give an example of a few of the nioh T space so you have the healthcare variables in the form of smart smart watches smart glasses Andy you're trying to get the data in related with the brain wave heart rate blood pressure calories burned and how you can monitor all these various data that is coming out of your body to proactively mom predict what's going to happen or what are the things that you need to take care of similarly in the case of oil drilling where you need to detect you want to detect oil leaks or spills of so that you can do a proactive maintenance in this context like the major metrics that people end up measuring all this pressure temperature air flow velocity on those kind of metrics and in addition to that cable providers they wanted to provide some kind of a way by which measure what is the quality of service that our internet is delivering for various applications how do you measure them so they might have the cable boxes and everything and they wanted to get the data collected from that box on senator or to the cloud to understand which area as an outage which area has a what kind of service that they are experiencing at all again these are all instances of the broader segment of IOT in the sense now let's go into a little bit more abstract what an I what the environment looks like so I would EGH environment consists of my rate of devices ranging from light device to all the way to the cloud so I have given some approximate characteristics about what a light device versus from advice was an H node and a cloud a light devices typically sensors they do only one functionality for example the air flow sensor is just going to keep checking for the airflow and send those metrics no simple to configure and probably as a lightweight protocol to communicate that metric whereas a smart device is typically as some low powered processor like ARM processors that are running inside and they are in general so multiple functionalities and in a broader characteristics fashion it expects some kind of a basic computation in a logic that you can push some things into it and the notion of some kind of limited storage for storing data and sending data out lightweight and our proprietary protocols to communicate from the devices then you have the notion of edge node which kind of aggregates a lot of data coming from those devices I mean from a either a light device or small device so it's typically multi-core based a couple of sockets probably around eight course it does some sophisticated functionality in terms of running multiple applications it can do complex computation logic and a decent amount of storage as well and again for protocol to communicate to the cloud it might be using lightweight protocols or even propriety protocols of course cloud has a lot of multiple missions and versatile functionality about what are the computational framework like machine learning libraries or even lambda functions everything you can do in the cloud and of course it does complex and generic computation logic on lots and lots of storage you can store the data forever or you can store the data as long as you want to retain them as well so this is a typical I the environment scenarios now there are multiple ways how the AI or the architecture gets deployed so one is you directly connect the light devices to a cloud so light devices have this simple endpoint where they are looking at some simple sensor data and then they are in doing some simple i shouldíve u protocols to give that data to the cloud if the data is lost the application is fine with it so this is one architecture the under that architecture is where you have smart devices connecting to the cloud where smart devices are like continuously collecting data they are doing some kind of processing on the data and before they sending it out so take the example of in an Hospital where surgeon might be using a monitoring for the heart rate that mission is powerful enough not only does my heart been monitoring it can generate some kind of alerts and if everything is fine in might not even send data only when some normality happens then the data is sent out so from that point of view it's smart so this is a kind of small devices to a cloud then you get into all kind of other architectures where your light devices with the edge where you have an H a greater and this is especially useful in a context where the age could be disconnected from the cloud for some amount of time take an example of a train where you have an H box finally you have a lot of sensors in the train which is collecting data about the engine and how its performing versus what's happening in each of the cards in the train and other things right so there will be an edge box which is sitting on the engine and trying to figure out how the train is operating but the train might not be internet connected to the cloud so which means the edge is the point where you take localized actions and remedy that action then after the cloud connectivity comes back and send it did this thing similarly you have the like smart devices with age as well and of course you can have another set of these things where you have hybrid set of devices some devices are lightweight light devices and some devices are smart devices all sending data in some form or other for crunching the data and of course then you have the multi cloud so now as you can see the for all it's complex so not only you have to capture the data from those devices you have to do some kind of data transformations and then data movement then process some de logic in order to observe some patterns in the data so like a normal detection outlier deduction or heavy thirds or how frequently there's something's occurring so you have to efficiently have us some kind of a data fabric that can capture data then process data move data to the cloud then also what do you call the process in a machine learning related algorithms on a heavy aggregation of devices putting the data together so essentially you need a some kind of a IOT fabric that needs the data to be flowing around so what are the requirements of that IOT data fabric so one is you capture data across devices on the edge i normalize or transform the data one of the challenge in data processing is 80 or 90 percent of the time goes in getting the data to the right format that you need so that is one of the important requirements of a data fabric and move data from the edge to the cloud and distribute for processing and when the movement occurs it's possible that you might have a disconnected operation where I cannot move the data because my connectivity is not working especially think about 4G connectivity another connectivity where sometimes people even though they may or tau all the 24-hour connectivity they might not use the connectivity because of cost reasons then sometimes you have to take proactive Dacians at the edge itself or sometimes on the cloud depending upon how critical the nature of the Dacian should be and process data centrally and push it down to H for example I have collected a lot of data from edge devices or a edge boxes now I'm do managing the data across all those devices now we need a way to compute those models then once the models are built then you push those models so that those devices become intelligent based on all the data that you've collected so far so you need to have some kind of a process data centrally and push intelligence to the edge then quick reactions to insights to mitigate risks especially at the edge and also because some of the IOT data could be very personal data so which means your dog secure the data while not only while moving but they're also doing processing time and today like the state of the orders in getting all kind of different set of projects different set of systems and everything not to glue everything together so ideally a goal to achieve is a single system that can encompass the entire data fabric so which is easy to manage and deploy as well and so I've given some requirement in terms of what are the three broad pillars that you need in order to realize the rate of fabric 1 is notion of messaging a notion of a storage a notion of for data processing so if from a messaging point of view you need very fast injection of data then he did need no data loss then you need some kind of security and encryption as the data is being ingested as well as moving around then you should support for millions of devices like especially in the IOT context you can have millions of devices or milli mirror several hundreds and thousands of stores sending data from other point of presence then it should be highly available and resilient and authorization to share data how to share data across multiple people who are working with the data to infer some kind of insights and the support for some kind of schema to identify what structure of the data is n and then you need of storage the storage could be at the device or the storage could be at the edge box our storage could be at the cloud it's able to write and bead with low latency and the durability which means once the data is given to storage it cannot lose the data unless it's explicitly deleted and it should be immediately available after writing the data a lot of storage systems data is available only after certain amount of time once the write finishes take the example of a Hadoop if you have unless a file is closed the data is not visible for further processing right in our case like it might be critical enough the rate of peas for example if for somebody's blood pressure shot up high you might want to immediately get alert into that right so when you have such applications when the data is returned immediately you should be able to pick up the data so of course the data at rest should be at the encryption and of course you need authorization first to rage as well and the processing there are a lot of processing that we've been used to that's crossing real-time processing and what type of processing that you need sometimes most of the time if you'd look at 80 to 90% of the these things about data transformation data routing data formatting and then then simple aggregations on top of it and some thresholding if this is aggregation exceeds certain this thing then send you an alert and then the API should be simple so that a lot of people can use it and the support for multiple languages each one has a favorite language like Java or Python and go then even WebSockets node.js then some kind of a state management so that processing can store the intermediate state and retrieve it whenever you need to and of course the SQL support is something that everybody asked for because everybody knows SQL because it is easy to write so that is when we construct a data fabric using something called Apache pulsar that has all the requirements that I talked about and what is pulsar pulsar is nothing but a flexible messaging plus streaming system backed by a log story if you are asking for one line what pulsar is this is what it is and how does it what does it do in this kind of IOT like environment there are three variations of pulsar pulsar has been made to run on the cloud and a massive cluster size I need pulsar can run on a single node with like a to summon cores kind of this thing now it also can run on a Raspberry Pi so so that we are using pulsar all over the that various points in IOT you can move data and process data as well as store data and distribute the data to other application that you like so since I don't have a lot of time to go delve into a detail of lot of the Pulsar aspects I'll just give you a overview of what other features that pulsar has pulsar it's completely durable in the sense like data is replicated as well as F sync to disk so that you never lose the data at all and it also provides multi-tenancy in the sense like you if you have one single flustered can have multiple teams working on it so which means let us say like you've take effective floors that are in one or a company might have multiple factory floors now each factory can see their Warren data but they might not see across factories data for certain reasons or the authorization purposes or whatever it is right so each factory you can map into multiple tenant so that automates the corporate can see the entire data across all the factories but each factory could be on tenant and each factor cannot on multiple application in the context of namespaces and you can have topics which are the streams of data that's coming from here several units of interval in the factory then then it also supports geared storage in the form of there if the data exceeds a certain size and also itself can handle all the data but if you don't want to keep the data in a hot data store instead if you want to it ages after certain amount of time you might wanted to store that in a cloud native storage like s3 or a Google Cloud storage or whatever might be the case right so it supports tiered storage their data after certain amount of time is automatically propagated into those tiered storage it comes with the data replication or across data centers out of the box so you can have one pulsar cluster on the edge box or one pulsar cluster on the cloud and he tell the Pulsar clusters each box a replicate the data to this thing it's one single configuration it can start replicating the data or moving the data essentially and which is ideal for an IOT kind of environment and seamless cluster expansion when you have a large set of data coming from several millions of devices you can keep on adding nodes to the pulsar cluster really expand without having to do any kind of heavy rebalancing down time or whatever it is you can continue to operate pulsar and you keep can do maintenance on Pulsar whether it's an operation then we also have notion of pulsar functions which allows you do to a flexible lightweight computations in the form of functions it's all like a lambda function this function can run on the edge box or it can be run on the device or it can be running on the cloud as well on the all pulsar instances and he does a multimodal way of deploying in the case of box itself or the device it's deployed within the JVM itself you don't so that you don't need extra resources not to run those computation logic so also also supports a lot of unified message modeling where you have the what cue model as well as pops up like model it provides high throughput and latency or it's highly scalable if you have millions of devices then you might wanted to make sure that all the data for each device is collected together so you can do millions of devices pulser runs are 2.3 million devices million topics that Yahoo itself so now where does the ball just get all the trends the fundamental impulse are is different from lot of this traditional messaging systems because he does what he called a decoupled architecture so it does the notion of a broker followed by bookie of the broker is the one which is serving data bookie is the one that is stores the data so because of the fact these are all separated you can increase the serving needs by scaling the broker and you can increase the storage need but it's just storing increasing the scale of the storage itself and so you know addition falser also stores the data in a notion of data segments so in a sense like each partition of the topic or whatever it is is chopped into smaller segments on the segment's are stored in an underlying storage because of that when some nodes die or whatever it is the recovery time is pretty fast because you do many too many recovery process because multiple nodes are participating in the recovery and whenever you wanted to query the data on a single partition you don't hit only one disk because of the fact the segments are sprays across multiple discs you can read all the data for partition in one single shot using multiple disks so that you can exploit a multiple disk bandwidth so data movement also can move data seamlessly from one data center to a two data center be and this property is the one that we exploit in IOT kind of context where an edge pulser box can send data to the pulsar running on the cloud where the cloud is getting lot of age boxes data coming from the over from the factory floors or manufacturing floors and getting aggregated there so so you it can produce at multiple different data center and consumer to different data centers or it can do it as active active with deployment or active standby deployment you can now all kind of you can even do synchronous deployment or asynchronous deployment so it's all integrated into a message flow a single system and also like a simple configurations to add and remove regions or adding devices or removing devices then similarly you can do asynchronous and synchronous replication as well so and I told talked about multi turn she provides where you have notion of a tenants and teams and within a team you have the notion of applications in this context we have one product layer safety II as a team which in turn causes of ETL as one of the applications and fraud detection is another application and within that you have the notion of topics it has authentication to verify one team member does not get access to other premium members data then you have software isolations where each team gets a quota you cannot have more than certain amount of quarter-sized so then you can even isolate them hardware fashion as well so each each team can have some amount of storage quota as soon the speakers like that marketing is ten terabytes whereas product safety gets seven terabytes now pulsar clients could be returned on several languages in addition to the WebSocket support for example in the case of light devices that I mentioned they might just support I should he be protocol soap also supports rest way of ingesting data in addition to all the other languages like C C++ Java Python and go so I mean pulsar has been running in production for four year four plus years on several companies and this is the largest that we have pushed it so far and this is just a handful of companies that have to put in the slide but around 100 plus companies have started using in some form or another now how do you process data in pulsar so one there are three aspects in which chickens process the data and pulsar unless consume data as it's produced so which is yeah I write my own consumer I write my own processing and then also provides something called a lightweight computer which is to transform and react to data as it arrives in the sense like how the data is continuously coming in you can do some amount of processing like in the form of data transformations ETL data conversion even routing then counting aggregations all kind of things like that then finally since we store the data in the storage Paulson itself has a storage you can query the interactive streams if you store the stream of data for a particular device how do slice and dice the data hey can you do like a week ago did this threshold for this device that is monitoring will its pressure exceeds certain value so you can query that the old data as well too so the lightweight compute is an abstract form of like a regular functions where your incoming messages functions can take multiple inputs and it can emit output output messages based on the logic that the expression functions how does the function looks like it's very simple you know it's just a drive from a Java function and write your logic apply logic and tell this is the input set of input topics that I need to grab data from and this is a set of our data that I have to write it in the output topic it just keeps going so it supports all kind of processing guarantees like at most ones at least won't effectively once it's controlled by line runtime by the user when user gives the many submits a function at the time you choose what semantics you want and remember this processing also can run on the edge there when you submit the function you have a rest endpoint by which you can remotely submit a function to execute on the device or even in the edge node in because the edge nodes might be running on a remote location which you might not even reach reachable to right so if as long as the internet connectivity and other things you can submit a computation logic to run on the edge as well so then finally you have query by using pulsar sequel where because of the fact already data is stored in the form of smaller segments and that data will be read simultaneously by a query SQL engine too poor answered a SQL query that you posted so now pulsar age I mean we can still have the separation of broker and bookie and the function running on a single node is requires like 100 to 200 MB of footprint and the runs like on two to three cords and fight all MB of memory function in separate GE JVM or you can run it on the same JVM as the broker itself on the pulsar and the device which has a 60 to 70 MB of footprint runs in only one core on 256 MB of memories good enough I mean this is rationally like what we have done with it Raspberry Pi and functions are run as a thread in the broker because you don't want to instantiate another JVM in order to run your computation logic so everything runs in one single process by itself now tying everything with the Pulsar together how pulsar can satisfy the needs of a fabric that we talked about as I said like this is a hybrid environment where you have seen pulsar running on a device which is running a function here like a filter function here which is not capturing the data from the device and the data is replicated using the giro application capability of pulsar into a pulsar age I mean well light based devices can use the WebSocket in order to pump data continuously into pulsar and then you can do some kind of data transformations on the edge itself before even data is moved to the cloud as well too so some kind of localized processing that you can do for example I wanted to get all the data coming from all the factory flows into a normalized form right so you can push the transformation function from your cloud into the Pulsar edge so that every data is and the same common denominator right so and similarly some other pulser function can look at some kind of aggregation function that is running so that it can look at whether some threshold as exceeded or beyond a certain range if it is exceeded certain range immediately you can trigger some kind of alerts so now once the data comes into the cloud now the Apache pulsar cloud version can distribute the data to for further additional processing like dashboards or even monitoring on bi and do a data distribution in terms for other applications to consume like of historical that are positive like data like which is used for bi and other things extra so that's all I had thanks for listening you have any questions yes a data for streaming oh okay data versioning so you could version the data it's not in built into that but there is a function that we have return special built-in functions that allowed you build the data versioning yes so that you can associate a particular device with a particular topic on the data for that topic will be specific to that device so which means all the data coming from the device will be continuously collected and if you change the particular data then you can change the data itself so which means it will be a complete log like thing it will be a complete log of all the data retain depending upon your retention period for the how long you want to retain the data now you can create all the device data separately to for some aggregate processing where you can take the data from the device of topic transform it and put term into all of them into one single this thing again that is all impulse are because pulsar function will take input from a topic and put the data back into another topic right so then you can store the data back into pulsar itself so you don't need a multiple external systems to dog bring it together you can do everything within pulse itself yes so I mean I didn't have time to go through all the details but remember all the lists that I gave up I know I gave in the first about POTUS pulser is that is the major difference if you want like let me this is I can go back to that slide so this is the one slide which has all the properties in addition to that pulser can run on raspberry pi and also execute on the raspberry pi so which is very small footprint so it is specially geared for a more on the edge edge kind of computations so these are all the various differences from Kafka's fall to and a pulsar was designed from ground up from there it was designed for systems where like rabbitmq JMS cuing kind of systems plus kafka like systems and they want to consolidate all of them into one single system it was designed basically that's where the two unified messaging model comes into play where you can do rabbitmq style queuing as well as pops up like half collects queueing subsystems in one single shot and in fact a lot of ideas from pulser has been gone on to influence other messaging systems as well like offset management absurd management you don't know how to manage any of the offset it's already inbuilt into pulsar completely whereas if you look at the history of Kafka and all offset management that got inspired after a pulsar was put into production at Yahoo then and pulsar also supports deduping so that the exactly ones there at least once and at most once do not suffer in performance they are in 10 percent D of performance of each other I mean I can go on for a lot of time but taking yeah yes yes you can be you can go back in time and replay from data also and it we can remember we can retain like six months or a year worth of data as well because ours is a separation of brokers and bookies right the bookies are the storage we can keep on adding storage as long as you want so efficiently we kvo proven pulser to retain more than a year worth of data a retains yes yeah you can service and you are it's all open source approach is open source you can download it and try it out [Music] so pulsar uses zookeeper only for metadata and again that is not exposed to the client at all in any way so you the point of interaction for Apache pulsar is only the broker which is sits behind a load balancer or if it is in a smaller age device right the pulsar one single process in makes the zookeeper also part of the one single process itself so that is why it's very easy to deploy on an H node yes okay oh thank you if you have any questions I'll be around [Applause]