reactive.community: Matt Klein Interview

Recording: reactive.community: Matt Klein Interview

[Music] hello everybody I'm Alexa crab Roe for the organizer of reactive systems meet up and here on location at nitro and we have for the first time in our meetup history we have really amazing stack written in C++ from lift engineering and we have Matt Klein nice from who a software engineer at lift welcome thank you so can you tell us a little bit about what does lift do with this software you guys work on and how did it come about and why is it so cool sure yeah and so we're talking about envoy today obviously so lift runs many services it's a pretty large service-oriented architecture I think the current numbers that we're publishing are you know greater than hundred services we do about two million requests per second internally and empowers essentially all of our real-time systems so whether that's locations ingestion or you know basically route-finding or ETA or kind of all of that so how on boy came about is when i joined lift about a year and a half ago on boy lift had pretty you know kind of normal service oriented architecture you know a bunch of services we were entirely hosted in amazon we're using al beads bunch of different kind of languages and random implementations of kind of different service call frameworks and a lot of the problems that people typically have kind of weather when they're running service-oriented architectures random failures the people didn't know how to debug you know people were generally kind of afraid of making service calls had a lot of conversations with folks about how you know they were going to put this thing in a library and like send it out to every service because they were too afraid to actually call other services so that was kind of the situation when i joined and you know fast forward to now we have envoy which is this out of process proxy and we use it for both egress and ingress for all service calls from every service and the idea behind envoy is that you have kind of a centralized piece of software it's not centralized it's centralized in the sense that you have a lot of code written in one place but it's fully distributed and all requests go out kind of a local envoy running next to every service and they come into another envoy an envoy handles you know essentially all of the common things that you need and service-oriented architectures and things like load balancing and circuit breaking and rate limiting kind of all of that good stuff yeah most importantly it allows us to get a really nice picture of what's going on in the system so that stats logging tracing um so kind how did this come about I you know I've been working on networking and distributed systems and operating systems for probably 10 or 15 years and I've seen how you know a lot of companies do this and you know we've kind of seen this progression over the years where a lot of the big companies who are lucky enough to use one language they typically are not polyglot there are typically using one language they typically build up kind of you know this this really nice library that does all of these things but you're forced into using that language essentially mm-hmm and unfortunately with a kind of a lot of the modern companies that tend to be more what we call polyglot they tend to be using Java and go and male Python and PHP and Ruby and kind of all these things it's very difficult to write a sophisticated library that can do all of these things in one place so before a lift I was a Twitter i wrote Twitter's front and proxy which is written in C++ 11 um so and that currently serves all Twitter's site traffic and I observed obviously and interacted a lot with finagle with it's you know kind of the service to service library that Twitter uses so envoy in in many ways is the result of kind of the observations of Twitter and actually before Twitter working in Amazon and worked on networking and Amazon ec2 not and just kind of observing how people deal with some of these problems and in some sense envoy is kind of a combination of multiple systems I wouldn't say that there's any component of envoy that is particularly novel but i think the packaging kind of how its put together is actually pretty novel and it's been really exciting just to see kind of the demand it's only been open source for kind of like a month and a half yes and and I've just been really pleasantly surprised with people coming out of the woodwork who were pretty excited like I've seen this up Twitter right right and that's like I grabbed it regards you know not every time but you know the reason actually I got so much interested I now understand that you know you've seen finagle and even kind of our GBM centric little bubble in our own big data and things like that so I've seen basically you know how much finagle helps yep Twitter and so now there's a company called boo in which basically took that and came up with linkeree yeah right which is basically I found kind of the language they are using is very similar all right all that talking about Roxy and and I find that I found like this is actually a very interesting way to talk about it because people previously you know they talked about essentially no HTTP or the token you know acha world like you know actors right but so I wonder who do you mean by proxy right and then you have l7 yeah right sure a little bit of all yep yeah no that's a that's a great question so kind of one of the design points of envoy and this is something that I learned from the system that i built a twitter is the proxy and by proxy a just mean kind of bump in the wire right it's a thing that's intercepting network traffic and in doing something look at your hopes it yeah yeah i mean it might be doing just sniffing meaning it might literally just be a bump in the wire or might be doing parsing and load balancing so so you have h a proxy is a proxy for tcp and HTTP traffic nutcracker as a proxy for Redis right i mean you have all kinds of different proxies and but what one of the mistakes i think that was made with the system that we've built a twitter at least on the proxy side is it was very HTTP mm-hmm and what I've come to realize is that yes you know most modern service calls are some form of HTTP whether it's rest or now we have G RPC from google right there's a lot of different stuff going on but obviously modern architectures you know they use a bunch of other stuff they use caches they use databases mm-hmm and you end up having a lot of the same introspection and visibility problems with those things that you do with HTTP so and by that I mean so it lifts for example we're using MongoDB hmm and on boy we actually have a MongoDB sniffer that's built into envoy that parses the MongoDB traffic as it's coming through and basically generates stats and we find that invaluable because now again when we access MongoDB from go PHP and Python we get unified stats and we get rate limiting and a bunch of other stuff that we've actually built in um so so that was a long-winded way of kind of circling back and at its core envoy is a network level proxy meaning it's kind of l3 and l4 by that I mean it's just bites in bytes out and then there's kind of a pluggable filter mechanism somewhat like what what they actually have in finagle um and then you can build kind of filters that do various things on top of that whether it be or rate-limiting or auth or and then obviously we have HTTP and then on the HTTP stack we have a second layer of filtering and that allows us to do HTTP level things just because that's that's so common um but at its core the idea behind envoy is that it's a bite level proxy that we can use to intercept all traffic nah and and I think envoy is most useful when the applications are essentially in a network jail and by that i mean the application can only egress go out through envoy and you'll only get traffic coming in through envoy and that gives you this thing the medically that you can trust that's generating again logging stats tracing all kinds of value-added features there at every hop and it allows the application owners to focus on business logic and we put a lot of that plumbing kind of at that base layer alright so basically we'll say you know by levels basically it will intercept all the bites and if you know somebody bothered to to parse them exactly exactly so out of the box envoy can actually just be kind of an ssl proxy replacement um so you know you can basically have a tcp proxy and then just do SSL termination on both sides and and you know it can it can have that basic function but then what the filter framework you could for example do rate limiting or you could do author or you could essentially compose a bunch of other things also please nice so and you know obviously another interesting thing for us is simple sports yes Cherie and so we used to see you know obviously a bunch of Java and Scala and then recently we see kind of devotes side gives rise to go like basically like Goa is kind of doing doctor and then suddenly is doing a bunch of other things yes and it kind of I think we see this also in kind of function programming world over scholar kind of is the most widespread I I noticed that when people want to do systems they often try rust because that's right that's another thing we just kind of build as functional but then the result of a blog post saying you know you know I tried response to programmer and then I had a lot of Tears yeah the best way to do it not treated as a functional programming search right and so I'm just curious right I mean a person didn't touch the plus + in a while yep right so why is the plus + what is your impression how do you compare with going wrong yes kind of policy yeah so I guess for for starters for full transparency I tend to be a pretty boring engineer III I like tried and true right you know so for for kind of a fundamental systems component like envoy when we were choosing a language you I I wanted an ecosystem that was already pretty robust right so that means you no good debuggers good profilers very robust library stuff like that so to be honest rust wasn't really on the table just because I don't really consider that at least its current state to be kind of production ready um you know what one of the things that I like to educate people on is I think a lot of people the last time they looked at C++ was when they were in college which is 10 or 15 years ago yeah and C++ 11 which was kind of fully ratified obviously four or five years ago and is in you know pretty much every compiler now he's honestly an absolutely awesome language and if you haven't looked at it I really encourage you to and even from a from a functional perspective you would be surprised to know that c++ 11 actually has in my opinion one of the best lamba implementations you know as capturing it's very explicit about how it works I think it's actually fantastic so you know I I personally you know I I don't consider myself a functional programmer but I do think that there are some great things from those functional paradigms that have actually made their way into modern C++ so you know I I think C++ 11 in my opinion is almost a perfect blend of kind of some of these paradigms so I would really encourage anyone that wants to kind of look at a high-performance kind of language with a great ecosystem to actually look at your post possible other guys do reproduce in suppose boss um I you know I'm not a MapReduce person i'm sure you can I couldn't tell you off top my head yeah how one would go and do that no the typically kind of you know in kind of reality at least you know I've seen like a first program very often it just has to be you know you basically work the whole collection right as opposed to kind of individuals and they have some kind of a map or reduce a fault yeah and then basically all you use basically producing louder yep right and I mean people suppose we'll not used to write like this yet but if the tools are there I mean there there are definitely libraries out there that do most of these things I you know like I was saying for me personally you know there's some functional aspects whether they be lambdas or other things that I think are really great and we actually use those in the Envoy coat on the style of the Envoy coats tends to be a little little little more imperative yeah but is the nature of network yeah right people accessible as possible that worked right since the beginning right yeah yeah and I guess the other nations probably that you do not really see big data because you can't even packets coming in right what did you about streaming like for instance let's say yeah man can you do give you know many primitive sister to interpret no 10 packets as a window and do something with them let's say the original streaming is all the rage these days rights right so can I do something like in like streaming processing sure sure yeah so on boy was was built from the get go to be fully streaming so meaning and by streaming he has built-in capabilities to do buffering because you often want to buffer a portion of things if you're looking at for example headers or any other things yeah but envoy in the filtering framework has the capability of both full streaming and also buffering and then essentially restarting so at least from the programming perspective we try to give people the ability to kind of do do build things whatever actually works for them on but yeah so envoy at the kind of the TCP at the bye Tim bye out layer has built to be streaming and also at the HTTP layer it's built to be streaming also or or buffering yeah cool oh yeah I won't fill up a little bit of all the simple choice as it reflects in hiring sure right so because I think they can only kind of the set of simple sports market was that people you know to tend to be with reducible plus skills you know to be very senior and kind of ulster it you know vacuous them out right so you have like high performance computing essential yeah right so so what do you find when you kind of hiring people to work on this project I'm just curious you know like how is basically hiring for c++ in this context is it easy as hard are what kind of people are interested in you know nice problem yeah that's a that's a very interesting question and I I think with c++ in particular there for better or worse there tends to be a lot of fun kind of around C++ right that it's this very scary thing that like only these you know crazy arcane people actually know how to do it and you know it is true that if you were to try to start working on C++ without proper mentorship it would be very difficult though I would argue that we work in an industry that is mostly apprentice based and that people learn how to be programmers by working with other program yes not everything right right so you know if you look at my current team you know I have three people working with me now and you know I think two of them had done c and c++ prior to actually joining and one of them has actually straight out of college and you know to be honest I I think C++ can be taught like any other language so with the right kind of a Prentice based setup I don't really think it's any it's any hard or less hard to hire for see applause applause I think it's just you want to higher quality engineers and if an engineer is interested and is not afraid of C++ because I think honestly of some of that Fudd I think they can be taught so I don't I don't view it as any different really nice and also I think the fact that you guys are in Seattle and if have Microsoft yeah store which is using a bunch of C++ right amazon yeah so anymore right right so I mean if you look like Microsoft Google I mean really at this point facebook I mean any large company at scale is using a lot of C++ still you know I don't keep track of the kind of the language rankings but I mean c and c++ even though we have a lot of people talking about a lot of other languages c and c++ are still very popular like yes yes sound reminds me the book about modern see no c plus bossa nova so cool something about this is not your father's see red like we kind of the Barkers now with all the side yeah yeah and that's and that's I think what what again people don't realize is that particularly with c++ with modern C++ I honestly cannot remember the last time that we had a memory leak in production right and that's just one example and what I find funny is that there have been so many times when I've been working with people who are writing systems on JVM or kind of other systems and once a year or twice a year there's actually a memory leak in someone's JVM project because of some complex cyclical dependency right can it's actually with modern tooling and kind of modern i would say language guidelines in c++ it's very difficult to actually have a memory leak right so that's it's just kind of one of those things that i have always found very ironic and I kind of leads me back to some of that I think fud just about C++ or I think there's this kind of like ethos or you know people view it as the scary thing but I would just encourage you know everyone out there that's watching to take a look at modern C and C++ because it's really fast the you know the tooling is fantastic the developer productivity i think is great so you know now i'm thinking that you should come back and give you this talk i would know i would i would say i would actually i would love to come back and do a separate talk just on c plus yes because because i'm gonna talk about it tonight but i'm going to spend like two minutes talking about sea posible it's just like what kind of ideas people are using like VI on the mac so like I lose my LG yeah it's pretty it's pretty varied now probably obviously there's a lot of people using them and yaks visual studio is no cross-platform visual studio is cross-platform a lot of people now are using the IntelliJ product it's it's the sea lion yeah you know so people who are generally familiar with doing all of that you know with with with their IntelliJ other things are kind of using that's easy I use an editor called slick at it whichever one at work makes fun of me for its this thing that's been around for 20 or 25 years but I've used it for 10 or 15 years and I loved it so I just keep using it yeah well the works it's funny because that's number one kind of skull and jawa right thing right and then of course you know cos versions for like Ruby and Ray beach well but that's but that's also part of it that you know C and C++ have just been around for so long that even whether it's vim or emacs or the other editors you know like all the tooling around it is is is quite quite good yeah and people are always surprised when I tell them yeah you can you can do autocomplete in C++ or shock no it's not it's it's not crazy look well I'm not you're actually curious I'm gonna poke around you know and voice open source right hey I'm gonna check it out so well i think you know we got really good picture so we looking forward to your main talk and you know we'll be happy to have you back at our meetup any time to talk about mother support great thank you very much that's my virginia [Music]