sfspark.org: Phil Gentry Interview

sfspark.org: Phil Gentry Interview

Recording: sfspark.org: Phil Gentry Interview

[Music] hello everybody I'm Alexa crab Roe of the organizer of SF spark and SF Hadoop meetups and here at uber engineering and we have a joint the top of SF Cassandra so we actually were talking about various use cases of the spike stack which is basically spark Cassandra acha Kafka and rhino select messes so we actually had spark and Cassandra connector today the other talk we had was with Phil Gentry from coffee made bagels and Phil is the senior ops engineer welcome Phil hey good to be here so first of all I want to ask you about this amazing company because I never heard about it it sounds really exciting candy like brief LOC lion what it does in weka sounds a good to turret yeah um so coffee meets bagel we're dating app we were only on mobile so yeah every day at noon you get a certain number of matches bagels that's what we call them um that are curated from our system you know we've got some machine learning things and you know we really spent a lot of time we trying to get really good matches so both men and women come to bagels okay yeah yeah every everybody want everybody everybody's a potential bagel everybody's back everybody's a bagel so yeah every day you log in and you get bagels and you you get to look at them and choose whether you like them or not if you're if you're a female the the ones that are always at the top are that are the males that liked you uh-huh so we really try to make it so that the ladies really get the best experience like guys you know you know you got to kind of put yourself out there first but you know it's it's really good particularly for ladies what is the geography of users I think we are we're definitely we're definitely most popular here in the US but we we have you know I don't actually deal with those numbers a lot but I can tell you that we're international mm-hmm and uh you can be a bagel anywhere just saying and in terms of us like you're in various places let's just in the area but yeah everywhere everywhere across the u.s. yeah cool so it's confident so you basically live at it develops nightmare because you inflict the doors on yourself every ever did not right pretty much yes it's a bit of it yeah from a system standpoint that's a that's a lot like what it is as a matter of fact yeah so how curious right like I'm in case only just one piece of it like how how do you handle this because it's a tough tough situation sure we have to pre warm load balancers mm-hmm make sure those are ready to go you can't just wait for you know amazon to just kind of like go by demand because by the time they respond mm-hmm it's over yes we have to just make sure everything's fast we have to be very careful about any background processes or anything that might be running and taking away system resources mm-hmm of course and we monitor closely what goes on and make sure that to make sure things are running well every day mm-hmm so yeah so this reminds me of another company all network called guilt right so guilty basically have especially the bicycle sell fashion discount and what they do like I think at 9am the basically announced net sales so the better you do the same Dawson themselves and but I think what they did they they went to Amazon because they want to scale up and down and and and they become super reactive and in our developer has keys on Emma's la cárcel the kind of it kind of changed the whole nature of operation so I wonder if this kind of modification changes your whole company and how you do develops and if everybody is aware like this thing is coming and like I need to be better already and so forth you know there's a little bit of that there is a little bit of that honestly like we are aware when noone is going to come it's it's not like there's a big like horn we're in a red light that's like deaths at noon but yeah absolutely we're aware of noon noon time and we do watch it but is it move it's noon local time right so I could ok so our biggest our biggest spike is gonna be here okay like it's gonna be west coast okay and then the other spikes like if that one goes well uh-huh we can relax a little bit right you know yeah yeah so okay all right Cassandra is new to our kind of meet up it'll it's the only letter from smack stack we didn't have you before right so like to us it's big deal we don't know much about Cassandra so can a kind of briefly outline how do you guys head like a senator what does it by you why is it good for this kind of things case okay so i can tell you i was not there when we chose Cassandra that's something I you know God but I think it has been actually pretty good for us mm-hmm the things that make it nice or that it is the horizontal scalability of it that is that's probably the biggest draw that I know of I just knowing because we do we definitely push the limits of postgres we used to use postgres we used to use a database called tighten that did not work out for us well and so post well it's wrong for me to say we push the long as a postgres we push the limits postgres on AWS and so we just kind of hit a point where we have to go on to something else so you use plus go before we still use it fruit for some of our stuff if we require that kind of transactional you know workload we still use postgres we're not one hundred percent Cassandra mm-hmm financials so some of this yeah honestly yeah you'd have to talk to to one of our actual back and engineers to forget that but but yeah we push as much into Cassandra I can tell you that as we can okay and you mentioned something like a lot of different issues right so again you know I'm firmly to Cassandra you know this kind of con Texas is new to do to us and so it is I find it curious that you talked about a lot of issues like like a better knowledge is it is it the typical Cassandra kind of situation right like the people I've spoke to yes as you grow and as you learn because the thing about container is it's it's it's configurable and I think the way that it gets configured changes a lot based on how exactly you're using it right writing time series data are you do you have a heavy right load reload what's a complication of your data you know there's so many different factors that go into it and how you're actually using it that it gets configured in a lot of different ways i think and and you do i have seen that story a lot talk to people and also just in reading and study I've seen that story a lot where ya people have to kind of work out how they are going to to use Cassandra once they start pushing it a little bit you know when you first set it up it's of course you know not that big of a deal but when you start actually pushing the envelope a little bit so this is interesting because you know I used to use edge weights you know like previous previous lives and in that the problem it was it was basically Union it was black magic right you could spend fifty thousand dollars and consultants trying to tune it yo you can find knowledgeable friends do it in one day right now so and so and at that point again it was about three four years ago the selling point for Cassandra was basically it's much easier right like you know you don't need to do this and somehow you know it's easier bicycle tune and it would you describe sounds pretty much like this black magic process why we should guess ya crucial parameters we should specific to your use case right so so is it so it's a I'm curious if that feels like that right so the comparison I don't have I don't have direct experience with that but the comparison that I've heard is that yeah it's still kind of like that but it's better with Cassandra but exactly it's me blows a great magic there yeah it's a great magic baby yeah little off white magic perhaps interesting interesting so so basically I'm curious because you mentioned you have the spike right so does the spike like it does this behavior means that you need to do this another bearers does it somehow like is your specific case what kind of tuning I would say does it require like you know for the most part it's really just a matter of making sure we have the capacity mm-hmm it's not tuning at this at the stage that we're at in with the growth pattern and the other things we've seen it's not a it's not truly a tuning tuning would be like oh well if we go in we tweak this memory setting or that memory setting then we see you know a better lower latency spike or something like that it's really just a matter of making sure that we're off to you know like we're actually scaled up high enough every day to actually handle it so at some point I expect that we will do more on the tuning level mm-hmm we just haven't got down to that part yet your other and so you guys are on amazon yes so how does this work for you is it is it like basically so you have you know VMs on amazon and then have cassandra on them does the like do sometimes we should create basically bare metal instances no but no well I guess maybe everyone through all yes in that we can maybe get some better performance out of them but no in that we would totally lose the flexibility that we have with Amazon's to just say we're just going to reinvent our whole setup yes and and move it over and and just try it in this whole different configuration without adopting all the cost of that hardware mm-hmm do you have your own dedicated instances there or like the do scale do you actually add notes Cassandra during the day or do you have just fixed a fixed amount of notes we know we don't auto scale we I hope we can someday but right now it's still a manually it doesn't take much people time to add a Cassandra node but it takes a while for that note to bring in yes streaming its data and the same thing when it goes out and then there's the whole repair issue afterwards so we're not we're a long ways still I think for more we can actually have that go up and down with their traffic that would be the dream mm-hmm but right now it's just yeah we're just looking at a fixed number of nodes and the instances themselves are they dedicated or are they shared like did you try different kinds of businesses we are trying different kinds of instances right now there's you know we have a few preserved that are that are the larger instances hmm there I believe those are dedicated mm-hmm don't call me on that then we're trying right now some smaller more medium-sized machines like in 44 XL's mm-hmm and we're going to see and those are unshared and we're going to see how those perform mm-hmm and then we'll go from there so yeah there's not a one-way question for that I would imagine that at some point you know because that's always a question on ews like yes you can get a dedicated instance and every once while or you can't where you can either shared but everyone saw with the shared you're going to have one that just doesn't act right probably because something somebody else is doing it you just have to live with the fact that you might just have to rent some time kill a node and and just yeah building it I'm gonna get the bad one yeah cool so I was saying kind of what's the most fun thing about working the camera oh man funnest thing about working with cassandra is probably you know really I just I just like it it's it works and it's just it's just the idea of it's fun to me you know that we have this this huge cluster of nodes working together and you know like that part I really like about it so though what it would be good to do right you know like to catch up with you maybe in six months and see how are you guys of all right because I think it like you you start with this you know yo Cassandra journey so let's sync up we know in six months and see how it goes yeah that sounds like a great idea oh yes okay thank you too much complete it thanks [Music]