reactive.community: Greg Silin interviews Justin Karneges about Realtime API
Recording: reactive.community: Greg Silin interviews Justin Karneges about Realtime API
my name is greg sola an imac organizer of the reactive systems meet up we're just coming hot off one of the sessions and here is the today's speaker Justin Carnegie's and Justin deewana may be present yourself yeah so I'm the founder of fan out and the lead developer pushpin the open-source reverse proxy for real time which I was talking about today awesome yeah we actually got Justin to the talk because I i saw i saw i'm going to different meet up and I felt that you know it's a good got a different topic for they react in under the reactive umbrella something we haven't done before and actually speaking of which just to put you on the spot how do you what does reactive really mean to you and maybe how do you feel you're the product you're developing are fitting in fitting in that okay yeah well so when I think of reactive I mean that's it's reacting to data and so you know we're moving to you know architectures that are streaming so you know you're dealing with data being pushed instead of you know being pulled or if it's being pulled it's being pulled in real time which is sort of like push I guess and so the way that you know pushpin fits into this and the stuff that we're working on it fan out is you know we're very interested in AP is and as these companies connect their aight guys together we need to make sure that we have data being pushed you know between these companies and so we need reactive development between different organizations and so I think that's the part of the stack that we deal with you know the sort of the edge between between organizations right so actually something you just mentioned right now you know you're interested in AP is and I think at the at the meetup you mentioned something that ap eyes are the future of technologies and certain tears of fury song comes to my nape guys are going to rule the world anyways and and I well maybe do you want to talk about that a bit yeah I mean well there's certainly more and more api's all the time I think ProgrammableWeb says there's like 14,000 now that are registered it's kind of wild I don't think we've really seen the amount of chaining of AP is that we that we could have you know I think in the future you know you're going to tap a button on your phone and it's going to go through like five different companies you know and may microservices at each company there's going to be all these connected you know api connected systems and so I think it's definitely a rising trend that's for sure you got companies doing API first where they make the API even though they might not necessarily need it yet for public consumption but they they want to be ready for that or you know they just they like that that set clean separation between client and server so the practice is growing but I think it's going to get going to get crazy I think in the coming years crazy w no operative world award and you know this multitude of different services and you know api's and platforms something that you could say helps you make sense out of out of all that is standard certain you mentioned during the talk to your a big believer in the open source standards I guess first of all maybe what open source what are the protocols that you guys using pushpin and second why do you think they're so important yeah so I mean I I wish there were more open standards I feel like we've kind of lost some of that I was I was big with XM EP and you know I help with the RFC on that wrote an IM client or you're going to ask when you said we lost you mean the community or yeah yeah so I mean I was with the you know the ex VP standards foundation and there was a big group there and so you know we were I say we because the community we were working together and even though I was you know authoring this client application there were other people running server or other components libraries and so on and we were all basically on the same team even if we were working on different projects and that was I mean ex MVP is still active today but it never really broke I never really broke into the mainstream and I got pretty close when Google adopted it that was pretty significant but then they got then they dropped it later and so you know I we had standards back in the day like the phones are the phones are standardized you don't have to be on the same phone provider to call somebody else email standardized you can be gmail and email somebody at Yahoo we lost everything after that you know social networking I am it remains crazy fragmented I don't know if you remember that what it was like in the early 2000s but there was you know aim icq yahoo and msn messenger those are the big four and we're we're trying to take them down or we were trying to at least have them use XV view and try to federated monks and then you Chilean if you wanted to use and you were to use something like that and but now we haven't we have a new new generation of systems now those those are all gone but we have even more new ones so we have you know Facebook or whatsapp skype and it's a whole new set of incompatible systems and and people aren't using the trillian type applications as much anymore they actually sort of just put up with it they have their phone has all the apps and you just deal with the fact that you have five accounts and so it's a very challenging problem because I think I think users have to care about it developers have to care about a company's have to care about and I'm not sure who cares about right now yeah well maybe part of it is that except for IM services which I agree it's you know it was good to kind of just like with the idea maybe and Mac nowadays to get all of your chats in one place you know like some of the more modern social technologies they still are so they did they try to differentiate and their services are so many different you know between the facebooks and the instagrams and a snapchat that I don't know if there is like really well those this common use case here those systems are different enough i would think maybe more about you know Facebook and MySpace now poor myspace nobody uses that but we don't have a face is that way never accepted my friend request I guess I never logged in but I you know I would love to see more competition I mean I i enjoy facebook but right now we have to do whatever sucka wants us to do and it would be nice if there could be an alternative really the fact that Google+ didn't really succeed means that if Google can't do it I'm certainly not going to do it so i don't i don't think it's solvable very easily what i would like to see is more companies that are in like dominating positions to care to care more about this it's almost like we need we need a don't be evil kind of or even more than that that not not just don't be evil but like go out of your way to to connect with the rest of the world and and it could be that just the folks that these companies aren't really thinking of that yeah well there's other standards the name escapes me at the moment but there's the the sorry i forget but basically there's a way to expose what does it do at all yeah okay anyways out maybe want to cut that part of the video but if i remember i'll email it separately but there is a way to to expose i think up you know objects in the API in a standard way if I forget exactly the you know things comes to mind was absolutely the wrong term there's a there's a like a standard I think like that certain API is used to like expose various like like hypermedia and forget anyways sorry there are certain standards that are emerging in the API community so that people represent objects and collections and kind of common ways wink headers and so on those aren't like full-on like you know application protocols like like a chat app would have but uh but it's good to see standards really any level to me awesome and just kind of a change direction for a second so we've talked a lot about during the meetup about the various technologies present in the real-time world you know you seem to have a lot of experience and have a lot of passion about that area but why push pin why as you know i mean ooh i'm an engineer I'm I'm trying to put real-time behind my api's why would I go with your product right so uh yeah the well I'll sort of start with sort of how it came to this the the reason I created push pin was was sort of this chain of thoughts so when I previously I was at live fire and we were doing the the commenting system with a live updates and and we had an API for it and so we developed the system it wasn't with tornado and had an API and but of course we were scaling and we started you know dealing with like lots of load and so you know we took advantage of you know cloud services where we could and so we were using Amazon Cloud front for example to delegate our our static pull traffic and so that you know reduce the burden on our back-end servers but we couldn't do the same thing with our push and that was kind of that was the first time I noticed okay it's kind of funky this is a different problem and so and I think around that time is when there were some companies that were just starting to offer sort of push as a service like pusher might have been around at that time but we were talking to a different company which I won't name but basically we were like in sort of a sales discussion there and ultimately you know the fact that and I mentioned this in the talk the fact that most push systems are into end we just just didn't fit it doesn't jive with an API strategy you know we want to make sure we owned our API and so transitioning to this other vendor would break the API and you have to use their library what if we don't like them how do we transition back I mean it took like Twitter takes years to phase out an API we don't want to be stuck with some vendor that we don't like that we can't phase out because people we don't pay or actually using the API like that's the trouble with a public API Jeff like folks using it that aren't your own employees so you can't really control like when they stopped using it and so so we just kept building it in-house and and so it really came from this perspective of I want to delegate real-time push load to a cloud but I can't because the right product doesn't exist and then I realize the reason we can't do that is because we don't even wrote the right engineering practices in place to allow a delegation in the caching world you know there are there's caching headers there's some there's some standards and in ways that proxies behave so that if you code your app in a certain way it becomes cashable and then you can use varnish or something or fastly but we don't have that in real time real time is just this proprietary mess nothing is interoperable and those standards aren't even in place so we can't even have a cloud to delegate to until we solve the standards problem and so so I thought okay I got to make the standard got to make a reference implementation so that's what push pin is and of course we also do run push pin my company fan out runs pushman in the cloud so that is actually the beginning then of a cloud service you could delegate push to and so yeah it is kind of a new way of doing things but it stems from this like just nobody has done this yet nobody's kind of like solved this well enough I think and so had to had to try awesome well hopefully you will be the next Facebook of real-time interesting way of putting it yeah yeah well actually one thing you mentioned is use is engineering practices and I wonder how being in this business effect I mean what are some of the unique practices that maybe exist on your team because of what you guys do well one of the things you have to worry about when you're building especially a cloud service is you know resilience you need to have high uptime and so I've you know we've definitely embraced microservices so we have like some 15 or so components on our server system that are just you know handling individual pieces of one piece crashes the other pieces don't you know aren't as affected or at all affected and so it's it's very defensive coating you know I used to take more chances when I was you know coding an app that was just running in one place like on a desktop or like a mobile device or something but when you have like you know multiple servers coordinating and you have thousands of people connected like you're going to make your own you're going to make a mistake and the impact is going to be massive and so you know simpler code you know microservices there's like less overlap of state those are those are the things we think about an architecting a nap and push pin awesome well thank you for your time sure thanks for coming over and x-ray will hope to hear more about you in the news awesome