data.bythebay.io: Yungjin Xu, Building domain specific databases with a distributed commit log
but it in smite so what is smite in Silicon Valley terms mine is a trust and safety platform for the internet but seriously what does might do exactly so smile is helping customers detecting all kinds of bad behaviors and bad actors on the internet for example we can help marketplace websites to detect for credit card transaction for such a charge backs as well as spam messages and we can also detect spam and harassment messages on social websites and for things like crowdfunding sites we can help them detect things like fake campaigns that a scam try to trick their customer to donate so we can tell all kinds of bad behaviors so that we can take care of the trust and safety side of the business so that our customers can focus on their core product experience and it turns out that to do all these interesting exciting things the centerpiece of our infrastructure is about counting which is all this talk about so what do I mean by that to do all this detection smart needs to answer a few very simple questions for example for a given IP how much activities have we seen so far for the IP or most specifically how many requests have received for that ID format IEP in the last hour or last minute right to make it more interesting what what if the IP is ipv6 which has much larger space or if we know a particular request is a forest sign up for website then we want to know how many sign ups has happened from the same single single IP in the last hour even more specifically if the sign up involves the email address then we can extract the email domain and ask how many unique emails have received for sign up from this IP for a particular domain from which we can tell if someone is automating a fraud campaign that try to build and create a lot of fake accounts at the same time similarly we can also create accounts for people who are sending a lot of messages that are very similar to each other which will use as a feature and signal for detecting as spam messages so this is one type of counting problem there's another type of counting which I won't be talked about in this in stock in detail but I want to mention here which is to do unique counts a for a particular actor that requires its own set of algorithms and then later our show that can be built with the same principle but I won't cover much detail here another note I want to make here data the con we're talking about here is supposed to be accurate count instead of a proximate of the count such as the IP number of IP addresses and the unique request from them so now that we understand what the problem is let's get started since we're very small start-up all we do here is that I'm I want to build something you know overnight so that next day we can just go start pitching our customers just it's just like building a plane during takeoff so and because we're building a database we want all these nice properties such as hoping sharted accurate and fought harder and etc in our system to do all these things you know we first started with a very simple solution we call it version 0 which is just using Redis and we divide our events into our time packets with expiry and all of a sudden we have a working database which is very easy to use because radius supports all kinds of common data structures and doing integers counting's in radius is very simple so let me give you a concrete example let's say at 4 10pm we get an event from a particular IP so we get we bucket that into a 15 minute bucket and increment account for that key during a before the packet in Redis and we we get account and then if the same time we want to get the count for that key it's very simple for the hour we just sum up the last the counts for the last four or five time packets and then we get the count right in the last hour so it looks like we're done this very simple counting you know this very easily shuttle because there's a string key in rattus we can have many as many rightist charts as we want and the problem is it's very expensive to do it because for a single count for the hour we need four to five buckets to store it and everything in Redis is stored in memory it's very expensive however fortunately when you first get started we get a lot of free credits for the cloud infrastructure that we use for google cloud amazon and as well deserve so we can forward a few expensive gears that's how we that's why we get started from from Redis however the key problem with this approach is that we're packing things into 15-minute buckets it's not that accurate and here's what we get so it's not very accurate it's kind of expensive and we can back up ready set up basis but durability is still issue because that is not a very easy to do and fault-tolerant is still an issue so but we have something working that can show and demo to the real customers using a relatively small effort now we want to improve it in the next version which enters Kafka so idea is that Kafka allows to have a durable sequential lot of messages why don't we just put the time stamped increments as Kafka messages so what we have here is that on we have a tuple of time stamp a key just like that so at certain time stamp we saw an LP and use that as key so now instead of having the clients doing both read and write directly with Redis the clients only need to write to Kafka and reform Redis and we have a separate Kafka consumer that reads this sequential log of messages like time-stamped keys and does increment so let's say we see this key at 225 p.m. so we increment that key 40 hour instead of having 15 minute buckets and what's interesting here that we can have a different kind of a consumer that's delayed by an hour so now later we can decrement it account for the same key all of a sudden now we can get the key for the hour and in time and have relatively a much accurate count for that particular IP what's interesting is about this approach is that we can extend it so that not only we count keys by the hour but also by the day by having another consumer that decrement the key a day later after we are first saw the key and can even extend it further by having a week count or even month count and because everything is persisted by kafka it's relatively easy to do but a problem is you cannot persist kafka forever if we want to decrement a key for a month we need to store up to a month worth of data which is very expensive fortunately what we can do is we can bad Kafka messages to s3 and have another cast consumer that is consuming from s3 files instead of Kafka cluster by by reading cuts through files directly as a result we can have the week and month consumers that does the job without having Kafka persisting or the data so as a result now we have a accurate count system that is charted using Kafka and we can we can sign reddit shards to couple competitions have a one-to-one mapping which make it very easy to scale scale out it's also durable because a combination of Kafka and s3 can save all the data forever but the problem is these things are still in RAM which is still expensive to run even though we have cloud credits and these credits expiring so you know we need to find a way to reduce the cost of that eventually we can manage the cost for it using our own money so now what we're looking at here is that we have three problems solve one now that we're not writing directly to rattus when you define a way so that we have a message for our rights and we need to manage the cost and fault tolerance so the man is autumn SED is relatively easy problem because when the increment fails before committing cover offset we may potentially bump the count twice and to solve this issue we can combine these two operations in one radius transaction by writing both the kafka offset as well as the count itself using one transaction and that gave us exactly one semantics now we only to solve the remaining two problems cost and fault tolerance because no lettuce is is something that either easy to get first clear started but managing a large cluster of Redis is not only very expensive but operationalize its it costs us a lot of hours to just to get it fault tolerant and you know we need to manage costs as or small startups that enters version 2 so there are few points here for us to build a next version of accounting database first of all we can no longer for putting everything in RAM so we need to put something on a disk but the same time we want to manage the latency so that we still have pretty good tail latency when we read from the database and finally you know it's already our third time my gradient data from exists there bears to a new version so want to make sure that migration is always easy and pain-free that's our requirements and I we still want to maintain relatively relatively accurate accounts because when we move from 15 million buckets to Kafka we improve the accuracy from up to 15 minutes of latency to just a few seconds that's how we are we when cover consumer may potentially lacked but that's up to a 15 seconds of latency so the few things that we don't need to worry about here for one thing is very important today's consistency we don't need strong consistency here that we need a count we don't need to count to be accurate by the mini seconds right a few seconds of a consistency eventual consistency is totally okay here and second we don't need to worry about like magical sharding using consistent hashing for example and move data around having a one cup cup edition correspond to one shard of your database is perfectly fine here for our purpose before we actually build our own database we did use and considered at least a few alternatives like my sequel a react and a few others unfortunately they're either too expensive to run or a very hard to operationalize at our scale so we decided that you know what let's build it ourselves so that enters our current version of the database so first of all we still want to maintain the radius protocol because very easy to write a passer where you should write a client and tech space so that we can make a tooling very easy but most importantly because our clients already know how to talk to rattus we want to keep that convenience for our next version database and then instead of using the commercial databases we decided that one we just replace a sink each single red is charged with impress s database like rocks DB which is open source by Facebook it's very powerful yet very tunable for different kind of workloads and we still keep Kafka for durability and replication and to achieve transactional support we use the same idea by writing both the count itself and Kafka offset using the same transaction that is directly supported by rocks TV and now you need to write server since we are you know a small start-up why don't we just use the most cutting-edge language which did and we prototype our server using rust unfortunately it didn't work out for for few reasons so eventually we switch to sip has passed using a few other facebook open source libraries tubular high performance server and mostly it's because what rocks DB is written in c++ so using the same language we get support for direct binding compared to other languages now we come to I will solve the few other problems and what's remaining here is replication and fault tolerance Kappa gave us replicated data log however the database itself is no longer a replicated because when you use Redis you can have ready spec ops not rocks DB so when you solve this problem and the way we do it is using another cutting-edge technology cubanelle use and this time is for real when we first started we decided to use kubin at ease when it was pretty much a i would say a beta software but we loved it and the way we use it is data we can use cout benetti's to manage a set of containers that reach would run a single single chart of our database and kuba Nettie's will make sure that we have a certain number of replications for our database if sudden host is taking down or failed it will create new containers on new hosts automatically make sure that we have the right number of replications and an interesting problem here at data rocks DB is a in process database that doesn't have any replication anything so we need to manage our own data replication and moving data around is actually a relatively hard problem however we're living in a cottage each major every major cloud providers have supported for some sort of now mounted a bit their discs for the empo in google cloud is called persistent disks so the idea here data instead of moving data around we can move this ground so for each rocks d big partition that crisp on to a cup competition we can write data instead of to a local disk we assign a network mounted is to it so when a sudden has two failed we cou benetti's will automatically create a new container on a new host by our mount the same disk from the previous host to the new one so we do that way we get data instantly and to a new host that is well supported without worrying about copying data around and the only time we need to copy data is when we want to increase the number of replicas to set up a new set of replicas and doing that is also pretty straightforward because we can just take snapshot of existing replicas and copy it to a new set of network money disks and start consuming from kafka because as we noted earlier those disks and databases already have the next kafka offset that we want to consume from so simple is that we can start creating new replicas in a matter of few minutes and a good feature about this way at this replication scheme that none of the replicas need to know the existence of other replicas because they read directly from Kafka and as soon as the cop car leg reaches zero who claimed that the new replica is ready to serve our benchmark we take so just a few seconds to consume about I think as 200k messages for each partition so it's actually pretty pretty fast in terms of getting a new replica ready for production use and finally we now we have a few replicas for the same partition so we need to a low balance to read across those replicas and COO benetti's provide a very powerful feature cut cluster IP basically creates a new IP address for each partition and for the client each client will run a proxy that is using iptables to manage the excusing replication across those replicas so that we don't need external open answer for any of the databases that we run the kind only need to know one IP prepetition now we finally get to this place where we have all the nice properties that we want for our accounting database and we build it using just a few are after shelf components that are open sourced and we are supported what's nice about this data instead of disputing this specific counting database we build it as a framework estimate and then on top of that we'll build a comp capability using the framework and just take a 125 lines I'm customer code we can get accounting their base up and running on top of this framework and what's nice about this room look at that arm now we can apply it to a lot of different problem domains in inside smack for lamp oh if we replace increment and decrement to set all of sudden we get using same idea we get a key value store and now this key value store not can not only support point queries rocks DB has a very powerful feature to support previous skinning so that we can do range queries in a very efficient manner another idea is that if we replace rocks DB or Redis in this schema and using solar as the underlying storage of sudden we can use solar and Kafka together to be your own search engine using exactly the same idea so that all sounds very nice but there are few problems that we encountered during the course first of all um you know there's a engineering our team that really love rust that's how we get started building those databases it's great language you know manage the memory in a very innovative way that is I think it's much better than cnc pass pass unfortunately since we heavily rely on Roxie be we want a sip hot spot space direct binding to it so we have to give that up and use the plus plus another issue is that um it's not fun combining the latest darker version and latest Cuban Eddie's and make them work together with a lot of production you shoes including temporary data loss because certain versions of these two software that doesn't work well together and we were not aware of that another issue is using the canary based Network Amanda disks is great idea and you'll still love it however it takes much longer than we expected to actually launch a new disk from a new host we would think you think you know the few seconds they work you can move a desk around but in reality takes per minute which is something that we really want to improve another issue is because we're writing to Kafka and reading from rocks DB we cannot guarantee read your own right one so won't you improve that as well and one idea is that when you read and what you have is a Kafka offset so instead of reading directly we can ask the database hey don't return any results and to cut up up to at least to this offset then I can guarantee the same really read for the right that we give just a few seconds ago so they have some future work that we're considering using the same framework first of all as I mentioned earlier a different type of counting problem is called unique count and there's a well-known set of algorithms called sliding hyper or hyper lock lock to to solve this problem but we can apply what's interesting you there we can apply this same framework and principle here and feel the same unique count database and we recently have that and running in production we're thinking of open source it because turns out that there are a lot of different scenarios where we can apply these awesome algorithms in production environment another interesting technology we're looking at is Kafka streaming which just came out I think a few months ago as a beta version software the reason why when you stood is because as we you can see we build our technology mostly based on Kafka and rocks DB which is directly how Kafka streaming is based on so we think we share a lot of similarities and maybe we can combine is to approach have a more scalable solution to our system all right that's that's the end my talk thanks for listening um I yeah so the question is yes so we between virtual mime-version to instead of being reddits we use in process database and radius as ouch sorry Roxy be is our choice yes yeah so the question is can we open source this code base yes we are actually considering open source our framework based on C++ so yeah the question is is this approach generalizable to other use cases and we believe actually as I mentioned earlier we after we build our database we found that the same principle can be simple can be simply applied to other domains of course inside our own team but we believe the same idea you know using Kafka for application and Roxy be for storage can be applied to meaning other problems and it's ready relatively cheap way to build replicated distributed database without too much knowledge about distributed systems sure actually I'm now where's you said you said Netflix is an open source project EB cash oh definitely i want to check that out but based on name i would say that's a bit technology based on cashing instead of a persistent storage of data i see as it definitely want to check that out thank you yes sparks dreaming so i think we saw me so stop streaming is it's happening something that we are we considered but the problem is you requires us to run is different set of frameworks so that's why we would like to check out kafka streaming which is more of a library than a framework that's less intrusive and we can maybe incorporate that in directly into ours in our own system yes mmm Troy what's the question sorry ah I see so yeah yeah so yeah definitely so the question is can talk about some of the attack algorithm that we use for detection for detection it's actually very interesting question so the way we build systems estimate is data which how to build a developed platform and what that means is instead of saying hey we give me give me your data and we give you the result give you results of yes or no right or wrong we build a platform so that it has individual building blocks for for detection and you can use them yourself and we even give you a github so that I can customize the behavior so what that means in practice is that for example I have discount in database that the count of IPs and signups and domain names give me the signals and we can provide a unsupervised learning algorithm as a building block for you to to know if if certain IP is also the email domains have a bad reputation and you get this reputation results and you can decide whether or not this is something that you log into and to say beauty on top of that we have mostly unsupervised learning models about about the detection of bad users for them poll and say hey a user if the user have a lot of Sun apps has is coming from IP or a dumb email domain or IFSP that we don't trust because other reputation data then we can decide maybe just something that into either automatically bad or something that people can look into manually so I think my point here is that we don't dictate on a specific algorithm that we use but provide all these information and data as a building block for you to either work with us to come up with the rules and models or you can customize yourself ah so we support actually support both so you can call us using asynchronous API like streaming and we'll give you a result directly but that can't because the synchronous that comes with a latency budget so we may not run through all the detection and classification algorithms but if you give us a message as in batch we may not give you answered directly but it takes few seconds for us to run through all the algorithm that we have then we can give you answer back using a web hook that way we can have more accurate shots but at the cost of a few seconds of latency yes difficult to ha yeah knowledgebase yeah exactly so that the way we look at it this problem is data we want to provide the building blocks for our customers so we take off the shelf and I'll p algorithms and and and look at the texts that people send us say hey is this message bad in you know because it contains harassment or spam and we assign score to it which we call reputation and then and people who use our system can take that result right they can either say hey you know we don't allow any harassment or spam messages in our system so as well as soon as we see a score above certain value we want to classify it automatically as bad so they take their results or yes right thank you cool all right thank you