reactive.community: Justin Karneges, Adding Realtime to your API

reactive.community: Justin Karneges, Adding Realtime to your API

Recording: reactive.community: Justin Karneges, Adding Realtime to your API

alright so I'm going to talk about how to add real time push to your API so if you've got a REST API and you know folks are pulling it over and over you should probably add some kind of push so let's talk about what that means so first of all real time is kind of an overloaded terms i'll define it now so for better for worse in the web community when we say real time we're often talking about push and not necessarily you know latency or you know time constraints or anything like that so you can think of real time being useful for you know chatting or games or you know news anytime you need the screen to update to show new content to a user sometimes this is called near real-time because it's often over tcp instead of UDP so it's you know might be lagged a little bit but it's being pushed at you as the main point and do you need real time you do the most popular apps have it these days you know if you've got a new a new type of product it's a great way to add some you know flash to it users are expecting it now what about real-time API is the most AP eyes are like rest they don't have push do do api's need to be real time they do some of them are actually already like Twitter for example and I'm a big believer of API sort of taking over the world and everything will be API connected and so if api's are the future of everything then they better have some kind of push right otherwise all of our apps are suddenly going to lose push you know so we need to somehow make sure that there's pushing our api's so briefly about me so I run a company called fan out where we help people with real time pushing api's before that I was a CTO at live fire dealing with their large-scale commenting system so I kind of observed the problems of developing a push system there and then wanted to make something that you know others could use I'm also just a big fan of like open standards and Federation stuff I was involved in X VP the RFC's years ago so all right so how do you how do real-time API is work so typically you have some kind of long-lived TCP connection you're not dealing with UDP it's pretty much all TCB based and what you see in the wild or you know unconventional uses of HTTP so things like an H to be stream for example like a stream that never in or you're using WebSockets some kind of non HTTP protocol and that's that's pretty much what people use so what you see out there is is I have these four categories HTTP streaming that's where a response never ends h2b long polling that's where you know the server hangs the request open but it does deliver a complete response and it's done websockets of course that's like the most cutting edge and then you also see web hooks and pretty much every API you see out there that has real time is going to be based on one of these four mechanisms or more multiple sometimes and so what I'm going to do with what i'm going to do in this talk is sort of go over how each of these work using real-world examples that you've probably seen and then get into sort of how you know how you can make something like this and how we can make it easier to build something like this so first of all HTTP streaming so you have this response it never ends the content is usually lines sometimes as lines of JSON sometimes it just lines of text depends on what you're doing server set events is also a very popular format because it's compatible with browsers so if you're using javascript and event source you can consume a stream that way service and events is actually a multi-line format but it still lines of text and so for example here's here's how to Twitter streaming API works so you know you make a get request you know just an endpoint you provide Oh auth credentials so it's very similar to how you might deal with their rest api the difference those that this request is never going to end you end up with a response that looks like this so you have you know there's no content length the transfer encoding is chunked and each tweet is a JSON payload that's all in one line so it's not pretty printed that way they can use new lines as a as a delimiter and so yeah so this is how it works in it and it's push twitter is not the only one to do it this way I found out the Gator does this we use Gator for our chat that was pretty cool if they're doing that too so what's good about to be streaming well it's HTTP so you probably understand it pretty well it's all so fast so unlike you know maybe like long polling for example there's no issues with sending a lot of data quickly to a recipient it's it's literally just a TCP packet once you've got it once you've got the connection open the drawback goes it's only one direction once it's open so it's not bidirectional like a WebSocket so if you are generally speaking you know the reason you might need to talk back to the server is because you want to listen to something else like me there's different data sources that will always involve opening a new connection if you're using H to be streaming but if you don't do that very often then it could be worth it to use HTTP streaming so long polling is a little bit different this is where the server has a delayed response so you make a get request and the server just holds it open maybe for you know 30 seconds or two minutes or whatever and then it either responds with new data at some point or a timeout some kind of time out data saying that you know there's nothing yet and then the client just opens a new request after that so this this is just straight-up HTTP it's just the server's holding the connection open amazon SQS uses long polling so for example if you want to you don't want to pull a queue to find out if there's new items if you want to just wait for an item you can actually pass this wait time seconds query parameter and that'll tell their server to just hang open for 10 seconds and you can probably set that value higher and then once there's data you get this you know you get a push but the push looks just like a regular response like they look identical and this is you know Amazon's XML format they're not the only ones doing long pulling either Dropbox notably does that as well and we also implemented long pulling a live fire when we were doing a you know especially the browser connections to the comments so what's good about long polling well it's restful you can actually literally document a long polling API probably with swagger or something in a little fit because you're not having to do something weird with some kind of long stream it's also sort of naturally naturally reliable because of the periodic requests so if you know you generally won't lose data with a long polling API the client will make a new request and everything will self heal your IP address changes it'll self heal so long polling is pretty resilient sort of without any effort whereas with something like a WebSocket you actually do have to remember to like track timeouts and send keep alives and basically essentially replicate a lot of what you would do is long pulling any way to have reliability downside of long pulling those that once you've sent something to the client the connection is closed or there sorry should say the request has ended the connection can still see open though but you can't send again until the client makes a new request so it's not great for continuous push you would not want to make like a collaborative drawing application like that has like pixel accuracy with long polling that would be horrible and then there's also WebSockets so this is a you know the new game in town relatively you know you get bi-directional message channel it's also support everywhere by now so I wouldn't worry about you know if WebSockets will be supported pretty much everything has it here's how blockchains API works so you know you make a request to this I envy employ actually not even sure what that stands for but that's where they emit their you know information about I guess a Bitcoin transactions and so WebSocket starts out with what looks like an HTTP negotiation but it would respond to the code 101 so that's how you know to go into this bi-directional stream and then once you're once you've got that bidirectional stream then you actually exchanged messages and so in the case of blockchain they have this you know sort of RPC protocol their designs of the client sends some JSON saying I want to listen two blocks and then the services here's a block when something happens that's their own kind of message format and since it also is encapsulated you don't need pretty printing or I'm sorry it's okay to have pretty pretty actually in the in the payloads so web stock is a very popular among Bitcoin companies for some reason slack's API is also websockets based if you read their docs I'm going to bring up see BICS later too because actually they use they use the push pin approach which I'm going to talk about next so great thing about web sockets bidirectional it's pretty much the best technology that you can use there's also no browser connection limits that's kind of a funny advantage I don't know why that's the case maybe somebody forgot to put the code in there but it's not limited like HTTP is so it's kind of a weird choice for api's though most people are still familiar with HTTP so I would advise if you did use web sockets as an API like as literally something you're gonna like tell like your own users to consume you might want to have something else like as well just so it's you know not to super weird you know you don't have status codes or header so all that kind of restful fun stuff is missing you kind of have to make it again and finally there's web hooks this probably easiest one of all this is where the server makes an HTTP request to the receiver no long live connections and you just register a URL in advance so github uses web hooks and so for example if you want to listen on for you know when a new issue is posted this is how you would do it you'd register a call back in this case I'm registering example.com / target down there in the config and then there's so there's multi steps here so you know github says okay we've created the registration for the web hook and it actually makes a URL to represent the registration there that hooks / 1 so you can actually delete it later that's how you done register and then when an event happens it does a post to that end point to that example com / target so web books are very popular stripe uses web hooks Facebook the great thing about web hooks is that it's so simple it's almost so simple that it doesn't even seem like you need tooling everybody can make sure make an HTTP request so you could just throw that right into your to your back-end code right now downside though is only servers can receive data so it's not quite as useful as the others it's also hard to test cuz you actually need a server to test it so you can't use you can't just like curl and point and see what happens you need to open up a port somewhere you sometimes have to set up like a reverse proxy it's a little funky alright there's all these choices long point streaming web sockets web hooks what do you use it's up to you really I think they're all valid for different reasons some companies use more than one like I mentioned Dropbox uses long pulling but they also use web hooks I'd say if you had to pick just one like you just couldn't spend time on more than one HCB streaming is a good compromise because it's very fast but it's still familiar if you want something literally today just do web hooks that's for worried about client connections later all right so how are you going to build an API like this and I'm not going to talk about web hooks because that one's a little bit easier but for the client oriented ones what can you do so there's a bunch of real-time solutions out there everybody's trying to solve real time push in a bunch of different ways you probably recognize some of these names on this slide most of them are in to end so you get a client library with them so you know if you use pusher you have pushers client library it connects two pushers you know domain name it would be an odd choice for a API and so with an API you have your API contract and you know it's something you own and it's not a some kind of secret or private protocol and so you need something a little different unfortunately than the kind of stuff you're used to and in fact when you do see an api out there that's real time like any of those like 20 or however many companies i mentioned earlier you know like slack and Twitter and whatnot they're all is all custom nobody's using something off the shelf and it's really just because nothing's been made to address the needs of api's and so and you really don't want to be coding on something like this just because it's it's so easy to get wrong you know I did that at live fire you have this like big stateful server it's got thousands of people connected to it you get one thing wrong and everybody gets disconnected like you don't want to be touching this thing very often so be great to just use something that existed but you're building an API and little too bad you don't get to use anything unfortunately though it's not actually the case so there are some solutions appearing now so I'm going to talk about pushpin that's the project that I'm working on there is another one though called stream data i/o which you can check out as well these are both solutions for AP is there actually also both proxying type solutions so the assumption is that you have some kind of API back in that you're going to hook this too and so but but API solutions are showing up so that's good so now I want to talk about pushpin and how you can actually you know use this to make an API so push pin is a proxy and it you know has some real time magic so it can handle a long live connections web sockets HTTP and you get full control over the interface between the client and the edge so this is what's unique about the project there's no pushpin client library the client library is you know it's something you can make if you want or maybe not well some API providers don't even have client libraries it's kind of your call you just get to decide what your API should be and then you can use pushpin to make it and it's also open source so it's on github you can check it out before i get into sort of how it works i just want to show you like what code looks like so here's an example of django there's a lot of magic happening in these few lines of code but essentially pushpin is sitting in front of Django managing the long live connections if you've declared this endpoint pretty much all you need to do is flag an incoming request as something that should be dealt with in a push way so for example set hold stream flags the request as a streaming connection that never ends you can assign one or more channels here we're associating my channel with the with the incoming request when you replied when you respond on this in point the body data and headers provided are sent as initial content to the to the client so what the client wind up seeing is stream opened with a new line and will be text plain as the content type but it'll hang open after this and then finally when there's data to publish on a channel you can just call publish in one line of code and send it down so how does this actually work though so getting into pushpin what we wanted to do here is separate there's these different roles when you're building an API different tiers in an architecture that kind of got lost a little bit with with the way push works so typically if you're building an API you know you have your API designer you have your back an implementer you have your maybe your DevOps dealing with caching layers you have your clients developing client a client code against the API that was defined first but as I mentioned earlier the way real-time push tends to be solved as these solutions are all into and so they it's kind of like it you know it encroaches on every single layer there and so we want to kind of break that apart so that people have the roles that they should have in this project and get some you know standardization and some interop in place so here's how i view the real-time API process so first you come up with your API you come down from the mountain and say this is how you will be accessing our live data and then your engineers get to work they implement this spec but you made the spec first that's the key thing which is different than you know something like socket i/o where they've already made the spec for you so the idea here is you you've made the spec first now how are you going to implement it now if your Twitter this is where the story ends because you actually just do the whole thing yourself in the back end but it would be nice if there was some of this work you could pass off and so that's where this pushpin proxy server fits in so you stick it in front of your back-end server and it handles all the like the long live connections so that your back and doesn't have to this means that you're back and can actually be a thread pool based system like Django like normally Django would be the worst system to do a push you know a push architecture with but when when you've got something like push pin in front then then it becomes okay also and we want to make an open integration protocol between push pin and the back end so the idea here is that you don't code to push pin you code to something called grip which I'll talk about in a moment but the intent here is similar to the standardization that's occurred in the caching world where you have like cache control headers you don't code to varnish or cloudflare you code to like caching standards or ways of developing and that way Devon says some some freedom there and then so once you've built this API using push pin for example now your clients can then speak to this API you built using the API that was defined and what's what's key about this is that the clients don't know that pushman exists again there is no push pin client library it's pretty much a detail of like you know the DevOps engineering decisions being made over there and the theory here is that now we have all those roles those the typical API team has their restored now but just in a push context and you also have the different pieces of architecture in place and there's some agility you can actually rewrite the backend say it's Django you want to change it to Rails sure pushman is still pushpin clients are still clients API hasn't changed let's say you want to replace pushman and move to a cloud service or something so we host a cloud service version of that and now could that could be a DevOps choice there's no engineering involved in that decision just like there's no engineering involved typically with changing between like squid and fastly so basically this brings push development to the same level of sophistication that we've already achieved with pull and that's great for DevOps because they can go to bed easy nobody's redeploying pushman constantly so that whole issue of like you know one line of code crashes and you lose thousands of connections that's not going to happen because you're not you're not messing with that your backends going to crash all the time but that's okay so a little bit about grip this was that open backend protocol that I mentioned the idea here was just to give a different name to sort of devender eyes deeper and it so that again so it can be a standard stands for generic real-time intermediary protocol and so the engineers target grip so you may have remembered on that initial slide with the Django example there was like the word grip in a couple of places maybe instead of pushpin that that's why and then this again gives DevOps the freedom to decide what actually occurs in that middle layer so to go through the flow of how this works you can probably imagine how this works already actually with the descriptions I've given but here's a sequence diagram just so it's like super clear so a client makes a request to the proxy the proxy forwards the request to the backend and this is pretty much forwarded as is so there's a cookie or authentication header it's all there bushman does add another header so you know that it came from pushpin other than that it's it's a it's a regular request and then when you reply you can reply with special instructions and that'll tell push pin to hold the connection open and so once you've replied the transaction between pushman in the back end is complete but the requests between the client and push pin remains open so if you had a streaming connection it's now open being held for updates and then at some future point your back and can push data to push pin and this is done using an HTTP post or 0q we have a couple of inputs that you can use their which then push me going to inject into one or more open connections for receiving and then that's basically it of how it works so let's get into the details with so grip supports both HTTP and web sockets now in the case of HTTP there's two modes there's I'm streaming mode and response mode response mode is long polling and they both behave a little bit differently so with streaming at instructions subscribe time you specify a partial response that the user should see immediately upon connection so they get this like initial status code headers some initial body have but then it hangs open at that point and then when you publish you actually publish chunks of body such that it you're just appending to this this stream that's already open response mode is much different in this case there's no initial response because with Long Point you actually send an entire response at published time so when you set up a response mode subscription you just specify the timeout response so what should push pin use if you don't send any data in time and then when you publish data you actually publish the entire payload status code you know body and all and so so here's how it like really looks so if your back-end responds with these two extra grip headers grip hold and grip channel this is staying hold this connection open as a stream bind it to this channel and here's some initial data that should be sent down and then when you want to publish you just post to this publishing point it's on a private port for pushpin so you wouldn't let Outsiders have access to this and you just say on this channel I have this content that I want to send to anybody that's an HTTP stream and that's how you publish data so I think the best way to understand this is to actually just just try it so let's see if I can you guys can see my screen here I like to do this in a vm I'll see ya can I do it's a pretty readable okay so I have push pin in here it's just a git checkout so I can just run it from the directory the most notable thing here to know is that it's listening on port 7999 as the input port so let me get a new terminal increase the size of this all right so i have so i have apache running on this box as well so here's here's the i'm actually gonna do a demo with apache and PHP because i feel like it's kind of you wouldn't expect to use that for real time push so here's the configuration for push but it's super simple the asterisk means any input so we're just saying everything that comes to push pin point seven nine and nine whether it's a HTTP request for a WebSocket forward it to port 80 on the local machine the over HTTP flag over there means to convert WebSockets into HTTP request which is another kind of fancy thing you can do with this i'm going to talk about that later but this is the basic routing setup and so I've got some scripts in the apache directory let's see so hopefully you guys understand PHP or at least you can figure it out so what this script would do is if you were to make a request this endpoint it would set these headers and then the body is just like floating text because that you can do that in PHP so for example if I actually request this on the local machine I get this which is what you would expect so you can see that those headers are being sent back properly so now what if we connect through the proxy what do you think'll happen so the request is hanging open now and what has happened is pushman interpreted those grip headers it also stripped them away so that the user doesn't actually see those that's actually another interesting thing about push pin is at the channel schema and the negotiation is all back end client doesn't even know it's participating in pub sub content-length is also gone it's been replaced by chunked encoding and it's just yeah it's just hanging open so then we can publish to it so open on a separate terminal here and you all you have to do is send a post request to the internal port which is 5561 so you can kind of see this line of code this is just a curl command and it's its publishing using that JSON format i showed earlier here's like the HTTP stream section of it but it's also at the same time publishing on like you know responsive and also WS messages web sockets so if there were any listeners of any type we're just going to send it out as a string with a sort of new line added of this yeah you can see it's dollar sign content and then a new line so it's going to print a new line so if I do channel is test so you can see it's streaming over there and so like how how easy was that alright so let's get back to the slides when I figure out how to get out of here alright so that was HTTP grip also works for WebSockets though and so the way this works is you the back end negotiates a special grip extension and so it's basically saying is you know I want the proxy to take over and allow pushing to be injected into open web sockets and then the back and can actually send control messages over that that connection between the proxy in the back end so unlike with HTTP where the back end only has one opportunity to speak you know a request comes in in the back end it says okay stream mode here's the channels done like I can't talk again because it's HTTP with WebSockets the the client or the back and can actually talk multiple times of the products they can add channels and remove them and so the way the differentiation is done is there's just the simple to character prefix if a message begins with a see in a colon it's a control message if it doesn't and it's a regular message or sorry if is an m cole and it's a regular message so pushman will strip off those two characters and if it's a regular message it'll ship it down to the client it's a control message though it will interpret it and for example one type of control message might be subscribed to a certain channel like channel tests and then when it comes time to publish the publishing is very similar to publishing for a stream it's just another format type and you can actually include more than one format like that that script I was using earlier publishes to multiple formats at once if different listeners should see different you know the data should be represented differently depending on you know may is not JSON that comes out of the web socket it's kind of up to you and also this crazy thing that Bushman will do is it will convert traffic from web sockets to http you might think this is a bizarre thing to do but actually it's it's pretty handy because it reduces the number of sockets between the proxy in the back end so let's say you've got like a million connections this way there's a million connections to push pin but there's not a million connections from push me to the back end but you can still have a web socket and so the way this works is that as messages come in from clients it's encapsulated into requests using these various events and the formatting is very similar to chunked encoding which I'll show you in a second and then there's no long live connections to the backend which also allows it to you to bridge it to kind of whatever you have you know if you already got like a thread pool based system that's like restful or something you can easily support a WebSocket with that type of architecture or if you have an API management system I Kong or something or three scale in there like you can just put this in front whereas those systems would like have no idea what to do with a WebSocket and so let's see the drawback of course with this is that the server only gets to speak when it's spoken to but it turns out that that that work so works out ok typically when you're building a WebSocket protocol you tend to have our PC and pub sub so with our PC the server has an opportunity to speak every time the client speaks and then with pub/sub that's usually sort of some spontaneous event and you can publish that instead so here's how the conversion protocol works we've created a new content type called WebSocket events and so when somebody tries to make a website WebSocket connection to an end point this is what gets sent over to your to your back in for the proxy just this open with a care to turn in a new line not literally those four characters it's you know it's just a two characters and then you're back and can respond with various events as well so this response here is saying okay I'm going to send an open event back then these accept the connection here's a text message with five characters hello I'm also going to close the connection and maybe you include like a status code in there so you can send as many as you want but you can only set it in reaction to an event that just came in so let's try this now all right so wow that's awesome am I missing borders on my wonderful geez sorry guys my terminals are misbehaving alright this one doesn't need to be big I'm just going to run push me over here Jesus alright let's get another window open okay so i've got another script in here let's see what was it called test websocket so this script just reads the request body and then sends it back with the right content type now this is actually enough to create an echo service believe it or not because if an open event comes in the way you acknowledge the connections by setting an open event back if you send a text message text message comes backs and close close and that close comes back so with that in place if we do this so there's actually a command a cool command-line tool called WS cat for web sockets and if we connect to the script connecting to a PHP script we have an echo service now there's no push here so let's do that next so there's another script let's see test a grip web socket so this one's a little more complicated it still got the still reading the entire request body and setting the content type and it's also echoing it back so we do have the echo serps but what it's also doing separately is if it notices that the request contained an open then it's going to make sure it sets up that grip web socket extension here and it's also going to set this sub flag so that after echoing it's going to send a control message to subscribe the connection to a channel called test so if i connect to this instead it's still an echo service but now if I let's say so you remember this this script here still use that same one and so you can see it injecting messages into that that WebSocket connection and then so there's I mean there's a couple neat things about this one is the fact that you don't actually need Apache running because it's not being it's not an effect right now push it push pins the one holding the connection open so you can actually still publish to it now if I tried to send a message it would I would lose the connection but as long as I turn Apache back on in time then the echo service still works so the back end becomes stateless here we have a state list WebSocket service with PHP so you can even modify the code and not disconnect something if you want to change the behavior all right now that you understand what you can do with the web sockets I'm going to sort of tease you with what we've done with Django because there's a little more magic happening here but essentially we create a middleware that parses out that events format and can also you know repack it on the way out and so there's this endpoint here and it's running a WebSocket system so if an event came in and it's wondering you know is it was an open request you can accept it and you can say subscribe to this channel and then you can return what you every return here is kind of bogus because it's just going to get rewritten by the middleware but in this way you can actually add a web socket in point in the middle of a django app and and you know handle our pc and handle pub/sub and it's all stateless so this this web socket context that ws value it's fresh every request so there's no there's no state here even if the first request is its opening in a second request as it isn't you didn't get the same object of you again could be on a different box if you have two instances of this django app the second one is handling the RPC call the first one handled the open request so it's kind of a wild yeah yeah why do you need state okay so so what you can do with this is on and it's not really described here but you can actually reply with state information like variables that Bushman should have so for example let's say you've got a WebSocket off process so you connect first and then maybe the user says here I'm cating here's some authentication packet and then when you reply to that saying you are authenticated at that moment you would also reply with metadata saying user is Justin yeah and so any future HTTP request that comes from the proxy will have that metadata on there so that you know it's that same user and and because the WebSocket is always tied to one push pin instance it's always going to be the one that sends the request to your back end so it will always have the metadata if there won't be this issue where like a different Bushman doesn't have the metadata because the request correct so if they lose their connection and they reconnect they have to re off which will cause you to then assign the assign the metadata again and yeah that's actually fairly new so it's not even in the Django library yet but uh yeah we want to make sure we can handle that another thing we do is we put a connection ID header on every request so you know that the same set of requests are actually associated with the same client if you want to like put that in a memcache or something and keep track of it but I think in a lot of cases you know it's best always avoided if you can and then there's always the metadata so okay and I mentioned that its compatibility with API management it's great because you can just put it right in front of whatever you're using and in the WebSocket conversion so oh and I want to show a real world example so it's big enough to read my light anyway so see BICS is canadian bitcoin index i mentioned them earlier they have a streaming API they also have a web socket API if you highlight up here you can you know see their docs and so and you can see here they did kind of a Twitter style API so they well first of all they have their own domain name streaming see BICS tat CA they have this a new line delimited JSON format and then they have these various endpoints like you know index which is like the value and then trades if you want to see what like activities happening in Bitcoin and they have these these JSON payloads and what's pretty cool about this is that they're using pushpin to power this so the back end is a larval app it's not really built for push its PHP that they have over there but they get they get basically a high quality API that you would expect from like you know Twitter or something but it's a larval back-end so pretty neat what you can do with that and they also have a web socket API because why not it's like not that much harder to do that with this system in fact they even have like resource oriented web sockets which is kind of neat so and then what we end up with though is that real time is not this special thing anymore you don't have to like it's not this crazy coding you know you don't have to like use event-driven frameworks or switch to know just because you want real-time this works with any web framework any endpoints for any reason maybe a query parameter causes something to become a stream it's like whatever you want you don't have to have split worlds we did that at live fire we had Django for rest and tornado for our streaming API two different code bases even though they were both Python they were still very different one was thread pool one was event-driven same guy did not maintain both there was some code reuse but you had to be careful you don't need to do that anymore with this system you can all you can put it in the same place and leverage is pretty much everything you've already built if you have an API stack so oh and a final plug about fan out we just you know we like to help people with real-time API eyes and a big thing about our approach is that our code is open source so you can always check it out and in fact you should I push me org and tell me what you're building I want to make sure that this actually works for everybody so thanks thanks guys so I don't know if anybody has any questions so the question is how does it scale so it's pretty easy to horizontally scale because the instances don't talk to each other and we've tried very hard to make sure that they don't need to like in the case that metadata situation it just kind of works out that they don't need to talk and so you can just add as many as you need we've done about I think the most we had connected to one instance was maybe like 50,000 connections i bet we could go higher we haven't done a benchmark in a while but it's easily horizontally scalable so you should have no trouble getting to whatever number you need to brothers mobile apps which are yeah so first what is pushpin built with or how's it made so it's a C++ app and it's it's actually a zeromq micro surface so it's kind of this crazy crazy project but I don't know if you guys know the mongrel to web server but it uses that for for input and it's actually set up with its five different processes so one handles HTTP input one hails handles HTTP output one handles proxying and one handles pub/sub logic and one is like just like a launcher that like runs the whole thing and so but yet so just basically a mix of C and C++ components that communicate via zeromq and mobile so typically mobile apps push well it depends on your app so there's a lot of Apple push and Google Cloud messaging being used to push to mobile apps and there's a couple of reasons for that I mean the main one I would say is because you can always reach the app like maybe your app isn't even running but you can use Apple push to reach it so you kind of have to use those systems but you probably wouldn't want to use those for anything that's highly collaborative so if you're building Google Docs or something like that you would you would want to have an open socket for that and I know some sophisticated apps what they'll do is when the app is open they'll be using like some kind of like long live connection to a server but when it's shut or you just want to tell tell user that something interesting happen then that's when you use push notifications so mobile apps tend to have a hybrid so there's a couple ways to deal with it in the case of a yeah when you have that conversion from web sockets to http there's actually a disconnect event that you'll get and then but in many cases you don't you may not care and I I don't think using TCP connection validity as a basis for presence is very useful because of timeouts other words or ghosting so sometimes it might take a very long time for a TCP connection to register as a cynic disconnected so it's often better to have some kind of heartbeat from the client yeah oh and another thing you can do with that so this is about detecting disconnects what you can do instead is you can actually detect subscriptions so that's another thing you can do so push pin will emit events whenever somebody at least one user is listening to a channel so if you have let's say you have a thousand people listening to one channel as soon as the first guy does it you get an event the next night here 99 you don't get any events but once they all leave then you get the unsubscribe and sometimes you can use that so if you have some like some yeah you're like searching for something there's some like back in processing you might wait for the unsubscribe versus the disconnect and the the difference is I don't think it's just a preference depends on what you care about if you think you might share them the channel subscriptions might be better than that better for that go guys thanks