sfspark.org: The Cassandra Odyssey w/ Phil Gentry
Recording: sfspark.org: The Cassandra Odyssey w/ Phil Gentry
[Music] hi my name is Phil I work at coffee meets bagel I'm a devops engineer part of the infrastructure team of two there I've been working with Cassandra for only like five months now so this doesn't get to be a talk where I like get up and I tell you how to do things because a pretty new to myself what it is is more of a talk about how we've been using Cassandra's some of the things that we've run into and some of the things we've learned and kind of how we've gone about addressing those things so first of all what does coffee meets bagel we are a mobile dating app we don't have a web presence but we're on iOS and Android are probably main thing that we do is we're focused on creating highly curated matches that you get every day at noon so you might get ten you might get 20 but those are going to come in at one specific time in the day it's not going to be like some other apps for you maybe swipe a lot and have like a hundred people you see we have an amazing group of Engineers where I'm just I don't know if that even belongs in this talk but I'm just really proud of the people I work with they're really great people really smart team and we use Cassandra as one of our primary data sources we do use some others but definitely we use Cassandra for a lot of things as I'm getting ready to talk about so use case number one what's happening at noon that's something we see on a lot of our t-shirts and things but if you think about what i just said that we do what happens at noon is we get giant heavy traffic spikes because all of these bagels or matches go out at noon and everybody is looking at profiles chatting see what they got today pretty much use case number two is what's happening the rest of the time which is going to be our matching algorithms are going to be running and using her data stores and doing as much machine learning and matching that they can do in order to make sure that people get the best matches they can get so our approach as i mentioned i'm not a long time cassandra user in fact the day that i started at coffee meets bagel I'd kind of had a vague idea of what Cassandra was but I hadn't actually used it and so one of the very first tasks that that I was given by another engineer was to spend up a new Cassandra node which luckily that was that went really well because it was a well-documented process everything was good but the funny thing was that he afterwards he said so don't worry because I must have looked worried but he's like don't worry in a month we'll all be Cassandra experts and we won't have to worry about this anymore it's like okay okay yeah so um five months later that's not exactly happened just yet but but we're we're learning we're getting better and we're still seeing a lot of growth so like all the time we're hitting new levels of traffic at the same time of trying to figure these things out so our general approach back to the slide is to you know when we we hit one of these snags we research just like anybody we theorize and then we experiment a lot and the reason that's in such big word big lettering there is just that it's really easy to limit if there was any kind of moral at all to this story which I don't know that there is but if there was one it would be to experiment a lot because you can you can kind of sit back and you can say well this should work this way and this should work that way and that kind of reasoning works so long as that kind of reasoning works and so the thing is we found it best to not like limit ourselves too much by that basically and to to continue to experiment try different things as you go along and then repeat so anyway so one of the first things I kind of hit up against is just the process of repairing her nodes it seems really simple the you know the the kind of naive approach is to just come in and just do it no tool repair on the command line but at that point our data was of a size and our cluster was such a sides that it we I'm not sure if those ever finished they still might be running in the background somewhere on our Cassandra closer i really don't know but it wasn't a good it wasn't a good thing for us to be doing to try to repair the cluster and the reason that's important is because we were having some issues and we were kind of locked into the cluster as it was we were afraid to kind of basically make changes to what we had because we weren't able to repair and does that make sense like we just didn't have that mobility to be able to test out new things so one of the first thing we did is address us repair to the repair issue we wanted to run regular automated repairs we basically when we tried to run a repair sometimes they would hang sometimes they would never return even though I think they actually finished okay and yeah I kind of skipped around a little bit so that was our first challenge the things we looked at to address that would be too well at first we just manually did the no tool to repair with the primary range argument on it on each of our notes so that works there was nothing really wrong with it except for it required our engineers to be up at midnight running no tool repair in a spreadsheet with all of our different nodes and it just wasn't very very good for the long term we considered using Cassandra Reaper we had some good we actually had some good experiences with Reaper it was actually good for us when it worked but the version of Java we were using at that time required some patching to the Reaper source and it didn't we weren't really sure how well that was I'm still not sure if a lot of people are still using Cassandra Reaper or not so basically just on the kind of complication level we decided not to use reaper the last option there that we considered was datastax ops center some of the newer versions of option arc can do repairs maybe it's not even just the newer versions but it might also be just if you have DSC which is something we've kind of gone we've been on the fence about a lot about whether we want to move to DSC right now we're on open source Cassandra we don't have anything against ESD in fact actually datastax has helped us a lot we're in the startup program and they've they've really done a good job of consulting with us helping us kind of work through some of these issues but that was an option and where we ended up was we just run a Python script basically we found a Python script on github the link is up there and really small letters Brian Galu what it does we made we made a few like alterations to it but basically it's Fletcher repairs in the smaller sub ranges and runs them independently so it just gives you a more granular control on your on your repair commands we run that out of Jenkins and it triggers nightly and basically we repair about a seventh of our cluster every day and so the idea is that all ranges will get repaired over the course of a week and so that's kind of what we're currently doing so that was the easy stuff that was the how do we repair our cluster number two going with the Odyssey kind of motif and the Cyclops memory management the real problem we were having is we're having Java heat issues garbage collection became an issue at a certain point we were seeing this sawtooth pattern when we looked at our memory management tools we saw long GC pauses like multiple seconds where things were just very unresponsive and so these things would basically you can see in that graph in the back that's actually a graph of a few of our servers during a time period the Java heap would just kind of rise to a certain point and then it would hit it hit a spot and our Cassandra cluster would have issues basically that that node would have to be restarted so we tried we went back to this experimentation approach and this is also where we talked to datastax and we tried a bunch of things we upgraded to Java 8 we started doing an O'Toole flush that one that one came out of nowhere that was just one of our engineers that had been on call and said you know I've noticed that when I want to get sleep at night if I just go in and do an O'Toole flush and flush all of her memory two disks on all the kissing it's like I get some sleep that night and so so that's where that came from and so we tried it and it worked well for us so we we automated it and it's it's kind of a bit of duct tape if anybody has any like I said I'm not up here as an expert if anybody knows what's going on with that that can help please talk to me when we get done here we switched over to the g1 garbage collector we started manually setting our heap sizes we upgraded or cassander 2 to 1 15 and we tried a whole bunch of other little tweaks and you can actually see up there on the graph that's actually over many months of time and as you go all the way to the left you can see this kind of chaos or these servers are going up and down you can kind of see then where they kind of line up a little bit those are probably rolling restarts we were having at that time and then you can see as we kind of started adding changing the the configurations a little bit more adding the the manual flushes flattens out increased our heap size raises up and now it's at a much healthier spot we don't have middle of the night calls that hey you have a Cassandra Cassandra node down so this is kind of an ongoing thing for us we're we're still experimenting with a lot of different things we have three we actually have three distinct key spaces I discussed to the two of the use cases earlier and right now we're working on getting those separated out into different data centers so that we can actually tweak those machines to be specific to that particular use case we're trying different instance types we're constantly bringing up nodes of a new instance type seeing how they perform and then if they don't perform well bring them back down again we're performing experimentation about whether we can use EBS drives or whether we can use instance stores on AWS everything we do is currently located on AWS there's a lot of conflicting information about whether it's okay to run cassandra on EBS so we're just kind of testing the waters and seeing how it works out we were trying different file systems xt4 versus XFS file systems all these things are just things that we've run across on the Internet somebody it's worked for somebody maybe it's not work for somebody else and we're just going to try it and see how it goes so issues that still remain for us actually we're getting decent decent performance and decent stability at this point out of our cassandra cluster the things that remain are we're still running that no tool flush that i mentioned that that was just a kind of an anecdotal thing that worked out still we don't know why that that needs to be done we do know that if it doesn't run we do have issues so it's going to keep going until we figure that one out one of our data centers right now is running constant compaction on the disk we don't really know why and generally we just want more automation you know it all of these things it would be but it would be best if we could just push button or even the dream would be auto scaling I don't know if that's a thing that Cassandra ever gets to the point of but I would love to see us being able to auto scale and yeah we're still working on it so with every iteration you know we we learn more performance gets a little bit better we get a little bit closer to that idea of you know the big win which is just this horizontally scalable platform where it just gets bigger and bigger and we just don't have to worry about it too much I don't know if that dreams really reality but it seems to be getting a little bit closer so PS we're hiring we're looking for back-end engineers back into engineering management managers senior data scientists and lead test engineer so if any of this sounds like interesting to you or you know anybody you know well you know coffee meets bagel we're hiring so yeah that's probably it I don't know if I ran through that really faster yeah does anybody have any questions go ahead sir so I'm just making to mention this before but how much traffic is your Cassandra cluster getting like reads and writes the question is how much traffic does our Cassandra cluster get as far as a reason rights and off the top of my head I don't actually know you know I know our our web app gets 15 to 20 thousand requests a minute during peak times but as far as throw put or number of requests I don't have that information off top my head sure yeah okay the question is under what circumstances are we using Cassandra versus other other backends that's one that one of our developers talk to you about I'm totally an infrastructure guy so I can tell you that we use Cassandra for a lot of our machine learning data but this the the difference of say like you can see postgres is on is one of our data or one of our data stores the the exact things that we storm post grass versus Cassandra I don't actually know the answers to that and we're trying to move everything else and then we use Redis feel like david has to be temporary please glass to search for well yeah we need to if we need to run complex searches on our data filters on HSN okay I other questions dish do we have a the question was do we have a reason to run the ranged repairs nightly yeah mainly it's because it's a it's a huge pain it's probably not a hundred percent necessary to get directly to the we could actually say only do that when we have an event say we remove a note or add a note or something where we may actually need that repair that might be a better way the the the general idea was just that at a certain point there's always something going on in the cluster at least that was our thinking that said there would all you know there's something we we've had one note just randomly died due to AWS death syndrome you know it just happens every once in a while randomly you know there's always going to be something going on so we just decided to just make it a rolling regular thing there's there's this configuration setting that you could put up on a table called read repair champ and that's supposed to advertise the time through here I don't know I know what version yeah so the question or the the the statement was that there's a there's a read repair chance configuration in Cassandra that basically as I understand it the there's a certain percentage chance every time a bit of data is read that when the driver goes and gets quorum or when Cassandra goes and gets quorum for that data then that data is automatically kind of repaired in that and that there is a setting for that and that's not one that we've actually used it but we're aware of it and it's it's in the near future yes it's something that's on our radar but yeah that's yeah yeah definitely on the repair side of things that no that was that's a good good point yeah yes sir the are typical cluster size at least on our main cluster is about three or 400 gigs I think there are typical note sighs and then I think right now we're running did Tina SH nodes don't quote me on that and so it's not it's not a huge cluster by any means okay question yes sir you were like way in the beginning okay the question is when when our customers see basically when they when the traffic happens basically right so yeah it's exactly that it's there's a big spike at noon and it trails off and then with the next time zone where it's noon you know a little bit further east it hits again trails off it's again and so it just depends on the yeah where you're at in the world even so ya know it's it's noon time in Europe so yeah there's it's not completely dead the question was how quiet is is it outside of a noontime and it's not it's not totally quiet there is it just it settles into it more you know just an even out traffic pattern like you'd expect to see in most places any other position I'm just curious if you tried using docker Cassandra the question is have we used docker with Cassandra and the answer is no we currently were using pretty good size machines and we have not we've considered it but we we haven't seen what the upside of actually using docker would be for us at this stage of the game it's something I like doctor and I've used it for a lot of things in the past but I can't think of a really compelling reason for us to introduce that into her Cassandra sec and use a case of requests you have a different they might say to it will that increase in a boat above knows what expensive the question is when we have using our moving our use cases in two different data centers won't that increase the number of nodes and therefore our cost the answer is in the short term absolutely because we have to spin up a whole new data center long-term the idea is that if we can basically we can reconfigure the same amount of actual resources that we have unless we're just under utilized or under provisioned at some some area so right now our main cluster our main data center is the same size as it was but the ideas get getting split out and then shrink things down to a point where it actually matches that's one of the one of the really nice things about AWS is because you can you can actually do that without having to buy machines which would be a little little different trickier yes ma'am yeah the question is a tinker right tickler tickler yeah the question was Netflix of this release tool called tickler that should kind of do that the answer is that happened I think after we had already implemented this so it's not impossible that will go back and check that out at some point but yeah right now what we have is kind of working but yeah that's a good point any other questions no oh one more they SED whatever you know that what are the upstream applications that stream because from Cassandra we that read from Cassandra right ok so the upstream applications mostly python-based there if not all Python base yeah so yeah just our AP is mainly are well ap is we do have some you know data exports for you know business reporting and things like that as well but mostly Python if that's kind of where you're asking yeah ok any other questions ok awesome [Applause] [Music]