Scale By The Bay 2019: David Winters, GDPR Data Cleaner: Mutating Immutable Data

Scale By The Bay 2019: David Winters, GDPR Data Cleaner: Mutating Immutable Data

Recording: Scale By The Bay 2019: David Winters, GDPR Data Cleaner: Mutating Immutable Data

[Music] thank you get on I'm David winters I'm one of the architects so the data science in engineering team at GoPro and and this talk is really a case study of what we had to do to be able to support deletion of data for big data systems something it's very prevalent these days especially with various privacy laws that have been enacted and will be enabling very soon so go ahead and get started here try to keep us on time I know it's a we only have about 30 minutes for the talk here so just real quick background on myself I've been fortunate to work in Silicon Valley in the Bay Area for actually almost 25 years for a lot of different companies and it's always with it really been with data related systems ecommerce back-end systems data warehousing search index systems those sorts of things and so uh the agenda it's actually pretty straightforward I'm gonna talk just a little bit about the Go Pro data platform just so you know the kind of type of data that we have and I'll talk a little bit about gdpr and CCPA which is the California variant of our privacy law that's gonna go into effect actually very soon here and then we'll talk about some of the challenges with the existing big data storage formats the immutability of them and then want to talk about some newer storage systems that we have available open source projects for big data platforms where you can't actually need to take the data and then finally I'll talk about an alternative kind of what we ended up and did a GoPro so that you don't have to basically rewrite or completely change your infrastructure in your big data platform but you can still support those two leaked requests so with that I'm gonna jump in here high level architecture picture of just the GoPro data platform at this point I'm not even sure we have many dozens of data sources some of the most important ones you'll see that in the upper left-hand corner the GoPro cameras so we do get a lot of really wonderful rich analytical data from our devices so we can really learn how people are using the product we can improve it even find bugs when various hardware and and third-party manufactured OEM you know parts and those sorts of things on the bottom of the diagram you'll see of the various icons for quick and our GoPro capture mobile apps and desktop apps a lot of people don't know that but GoPro actually creates a lot of great software it's actually all free and you actually don't even have to use a GoPro camera you can actually use it with your iPhone or your Android device to auto edits and then manual edits the GoPro quick application is excellent for that we also get really rich e analytical data from those applications and again can improve the system is the applications through that on the top there we do get a ton of social media data these days go from the GoPro we're very fortunate is a very well-respected brand we do live sentiment analysis and try to understand exactly you know what is the feedback on our products and in various promotions and then on the right side is more the egress while pulling insights out we use a lot of spark notebooks for visualizations plotly there's a course you know the marketers always love the CRM workflows and systems that send advertisements and through email and mobile push messages and obviously user segmentation those sorts of things a little bit more specifics here but I kind of covered then the last slide so I'm gonna actually move through this one so we can jump into the gdpr basics I'm gonna you know how it is there's always these legal disclaimers I do want to make sure everyone knows and I'm not advising anyone on gdpr or CCPA or you know practices or anything and in the things I talk about here are really for general background purposes it's not necessarily doesn't necessarily reflect business practices that we use and you should obviously consult with your legal counsel for anything related to data privacy laws they are they are very challenging to interpret at times so um what does GDP are and CCPA um so these are two laws that one went into effect last year May 25th 2018 the general data protection regulation it's technically an EU law but pretty much if you do any business with an individual or consumer who lives within the Europe you know a country within the European Union it pulls your company into it pretty much CCP at California consumer Privacy Act is California's version of it very similar but there are some differences obviously once again you should really consult with the legal counsel for those things but the the important parts are this there are various rights that consumers and individuals have two of them which you hear a lot about is a subject access request which is a user can go to a company go to an organization and say hey I want to know what you know about me and you have to provide all of the records there's also no Rachel brought in a ratio request which is a in a consumer comes in and says hey I want you to leave all my data there's different various means and how linking you know communicate with you and contact to you and you know legal timelines and so forth but that's the the basic gist of it and that the the challenging one is really the oratio request oftentimes because these big data platforms that almost every corporation has these days they were really built so the data is immutable it's not really easily easily deleted so we're not to talk about that next so we're definitely not going to go through every single spark or high file format definitely there's many dozens of those but I wanna talk about three popular ones here Avro which is a row oriented storage format and then Park Hae and Orsi which are columnar storage formats so when you start thinking of a Verrilli Cicero oriented so if you were trying to delete rows it's just you know a block of bytes there essentially that you would need to be a part of that stream of bytes the column inner ones are K no RC are much more challenging they store their data by column values so that would mean if you have say a hundred columns are sown your table you're gonna have to pull bytes out in one hundred 100 different places or so in each file just to delete a single record so that definitely is problematic and then all of these file formats through the years they've been around a while they've been utilizing lots of performance enhancements so they do use a lot of compression they're actually very intelligent about their compression sometimes it's block level sometimes it's page level they now have index structures to again improve range scans and so forth they page level statistics so that a lot of the various computations you try to do and group by statements and so forth are very quick but what it really comes down to is those files they kind of stand on their own and you can't just pull out you know a few bytes here or there you have to rewrite the whole file so it definitely makes it much more challenging to try to do updates and then when you take a look at the underlying storage systems that they use historically or originally they use HDFS the Hadoop distributed file system now that's sort of a meta file system it has these huge blocks generally anywhere from 128 megabytes to 256 megabytes sometimes even larger and that makes it again even harder yet if you're trying to just change just a few you know a few rows here or there you're writing there some pretty big blocks of data over again and then a lot of companies the last few years have moved more cloud oriented and so that means they're using object stores that are provided by their cloud provider Amazon provides s3 which is obviously used by a lot of companies that that's even more challenging there there really are no blocks at least externally to the in developer it because it's a black box internally there there may be and so what that means is you truly do have to rewrite the whole file if you want to make just a single small update so the takeaway again you know we already talked about the data privacy laws they obviously require us to be able to mutate that data and now we take a look at the big data file formats they're designed to be immutable and so the conclusion is obviously you need to have a mutable big data storage system the good news is that the Big Data community has has been thinking about this for a long time and so there are several options available these days you know some of the reasons weren't even always related to these privacy deletion requests if you did any streaming ingestion of data trying to do that with these big data infrastructures are very challenging you oftentimes it often times end up with many small files which is definitely gonna kill your performance over time if you have things like slow changing dimension tables where you have you know a reasonably good size table and occasionally a few rows change you don't want to really have to drop table and reloaded every single time it may work at first but as your data continues to grow you're gonna run into some bigger challenges so again an immutable data storage a format would help here data a statement and this is a basically the perfect case for slow arriving data a great example of this would be social networks you may have made an announcement and that announcement your company did on on say a social network platform and yeah you may get 90% of your data within the first say 24 to 72 hours that it may take up to a month sometimes even two months until you get that final we'll say that kind of that long tail there that that final 10% or so of data and oftentimes that data is your very valuable data it won't just be instinctual gut reactions to a some information you sent out there maybe it's related to a new product you have it may actually be data that comes in later from end users that actually bought the product have some very valuable feedback so you want to incorporate that data and once again you don't want to have to drop all the data again and then reload it just to get an update we've already talked about the privacy clean ups obviously needing to be able to me to take the data and then there's also um compactions I talked earlier about the small file problem which is a real problem with big data it's gonna kill your scan speed lots of i/o you know reading these small files you don't want that so you would like the ability to also be able to compact those small files into larger files and once again a mutable data format is gonna help you a lot more with that several recent open source projects that support mutations on the data storage Apache hoody which came from uber has been supported a lot by confluent recently the kafka the company behind Kafka it's been around about three years pretty mature product works with both hive and spark data bricks Delta Lake which started off as a proprietary feature from the data Brooks spark service and then I think it's about six months ago they open sourced it so it's a partial open source scene right now they haven't put all their features in it but it's definitely very promising and we'll talk about a little bit more in a few seconds here and then there's the Apache hive acid transaction tables which has been around about four years so it's very mature but it is really built pretty much just for a hive not really for sparks so far and then you have some newcomers like a patchy iceberg which is from Netflix and has taken more of approach more around query planning and execution right now but they're actually working on mutation support also so when you take a look at this and I'm not gonna go through all these tables these tables are pretty large but it kind of lays out a lot of the main features here and I actually color-coded that green for things that were good for our use case and then read things that were basically bad for us and the black is kind of neutral it doesn't really have an effect on us I'm gonna jump to this summary here well what really came down to is all of our requirements were not met by any of these open source projects and our requirements were at the top there we wanted the support obviously for some updates and deletes it could be through an API it could be through a Java API or Scala API or can be through DML which would be preferred which means you could do update statements and delete statements directly in the sequel now definitely be a great benefit for our data scientists which are going to be more prone to using sequel and a little bit less using AP is park' file support since about 90% of our data is stored in that column in a format I have sparked to have four down there but we're basically just saying a modern version of SPARC a recent version of SPARC manual or automatic compactions once again definitely a problem with all Big Data platforms you want to be able to compact those small files into nice large files to improve your scan speed I kind of already talked about that nice to have the DML support and then the concerns and this actually became some of the largest reasons for why we were not able to move to any of the the existing open source projects was the data migration time often times there was nobody migration path so there is additional metadata that's stored obviously to be able to track these mutations sometimes it's stored directly in the data files uber or excuse me hoody does that with storing it directly in that park' files definitely problematic cuz you have to rewrite all the parkade files some of the other systems just have other outside data sources they either use key value stores or relational databases or even the file system sometimes in JSON files but again all that metadata has to be extracted has to be populated you need some way to migrate the data and so we ran into either there was no data migration no path really at least or we'd have to create one or it was too time-consuming or just flat-out failed and then there's the insert and query performance overhead obviously there's over a head here you can't just go and read from the files any longer there's gonna be some reconciliation you have to do with some some external metadata so since all of our requirements weren't met binding these open source projects and and we also didn't want to have to completely change all the infrastructure for our data platform we decided to develop our own data cleaner tool which would allow us to basically do these deletions and keep our same infrastructure as it is so we talked about what we developed so our requirements were almost the same as I talked about before for our data cleaner project but they were also just slightly different in some cases they were a little bit narrower some cases are broader I'll just kind of go through them so we decided after we took a look at things that we wanted to support more than just park' files we do have about 90% of our data in park' we do support some other formats so we we definitely wanted to be able to do that we also wanted to support all the data types all excuse me on table types we we use pretty much everything under the Sun we have a mixture with some spark tables we have some hive tables we have partitions we have non partitions we have external tables with managed tables so we need to be able to support all of them we decided that as much as we'd love to have update support we could live with delete only at this point and then we did one a you know modern version of spark we were in spark 2.4 so we that and then the bonus was if we could have the ability to least do manual compactions would be fantastic so some of the things that you know kind of crept into our requirements and our thinking as we started to do some some proof-of-concept work and some early development prototyping is we we decide pretty quickly we we certainly didn't want to have to maintain all sorts of code for every single high file format that would for one thing it would just be redundant there are already platforms such as spark that already do that and so that made our decision to go the direction of sparse excuse me spark sequel for all their operations now when you look at spark sequel there is only one way to really overwrite data there is an insert overwrite command and you can do at the partition or the table level so that kind of knocks that off our list well we can support both partition tables and a non partition to tables it also worked after we did some tests and you did have to get just the right spark version because there have been some zeros that were fixed in time but it could also work for both managed and external tables and then the the best yet yeah it worked for both har excuse me hive and spark tables so this is just a very simple to give the you know folks especially folks are familiar with spark in the spark sequel it's a very simple example of what you can do with sparks equal to basically implement a dilly using an insert overwrite so in this case have a very simple employee table I'm using parquet again what we ended up and developed you can actually use anything you want but this example uses parquet I used snappy compression I decided to partition this table by a date and in our columns and then it's an external table since I'm supplying the location insert some some data here or my childhood favorite the Flintstones characters and then here's what you can do to to and this is actually working code you actually copy and paste it into a notebook or into a spark shell this is how you can use an insert overwrite to implement a delete and spark so we start off here we have to fetch the partition so you do have to do it one partition at a time so in this particular case I'm touching our partition for the date of 2018 Oh 802 and the hour of the midnight hour essentially zero zero and then there's some debug statements in there but I take that partition the next thing I want to do is you have to filter out the rows that you wish to delete in this case I'm filtering out any user that has basically Fred Flintstone's email address or this guy Ted's email address and now I have my updated partition now this is where sparked a nuance to spark spark is intelligent and it knows it doesn't want you to just pull a partition from one of its tables and then write it right back because it knows you could end up in an inconsistent state if the task fails and so it's Park does here is they force you to basically write it into a temporary table which is not a bad thing it's not a bad thing for one thing if your operation or one of your tasks fails halfway through you don't end up an inconsistent state and you can rerun them the operation but it's also good because it gives us the ability in the future to archive data as we're deleting it because even though we're deleting this data and we do want it to go away forever we also do want to be very careful at this data and we wanna make sure that there's not a mistake made and so in this case is a standard process for a lot of you know organizations can be you do your deletion then you verify the deletion is correct and then you drop your your archives and so in this particular case we're not going quite that full route to the archiving because again I want to make it a simple example but we have we ready the temporary table after we write the temporary table we're gonna read that back essentially into memory and and we actually have to use for this very last insert and overwrite the cleaned up partition that comment there you actually have to use spark sequel itself to do the insert overwrite it's just a nuance to spark sequel there's not a Scala or a Java API to do it there's lots of jurors filed on it but so you do have to kind of shift gears a little bit wait to go full spark sequel at the end there and do the insert overwrite but it actually works so what we did next was obviously that's a very hard code example that certainly isn't in a work for the well huge number tables how we have and so this is a class diagram of what we ended up in and came up with so we created a facade at the very top there user a user data cleaner class and the idea for this was its responsibility is to delete all user data you give it a list in this particular example on the right hand side you can see it's given an email CSV file so that's gonna be a list of email addresses it could be other IDs also obviously email address is not the only way to identify a user but this particular case in a use case that's pretty popular is email address you also give it on the left hand side we'll see a tables configuration file in our architecture it can either be a JSON file or a type safe config file and that describes all the tables so you list out all your tables and we're gonna take a look at example that next and then there's a series of delegations that occur the user data cleaner instantiates a hive table cleaner the hive table cleaner depending on the type of the table if it's a partition table it will instantiate a hive table partition cleaner if it's non partition tour just a good old straight table it's gonna instantiate the hive non partition table cleaner the nice thing is we learned after we did this was as we were doing the deletion it was a great opportunity to do the compaction so we could repartition the table at the same time so this say we had a table that would end up with every single partition having tens of thousands of files at that point we could take a lick a look we could analyze the the size of the data and we could we could be a basically repartition it so maybe make it you know a thousand different files and so that's what that hive table ready partition areas at the bottom is it's essentially a delete with a null filter it just filters out no rows and it just rewrites the files in a nice compacted format we also use a lot of HBase echo Pro and so we wanted the user data cleaner to handle all big data in this particular case so it also cleans HBase HBase is a transactional system so deletion fleets are natively reported so that's very easy here so just a few examples of some of the JSON configuration for some of the tables this is a like I said a JSON version of it you can see we defined the type hive in this particular case the name of the table of the database some things like partition columns that's something we could get from the hive meta store will be a feature that we'll work on in the future to just reduce the configuration you can define your ID column which is how you're gonna identify the row that needs to be deleted we later at it actually support for compound keys also and then we actually even support different types of encoding so you can do normalization of email addresses trimming off whitespace lowering it to lowercase text and those sorts of things and then we also have a type safe config version in this particular case you can see it also supports the HBase tables then also have some hive tables listed here so the summary is you know what we developed was we developed a cleaning Andry partitioning framework on top of the existing big data infrastructures that everybody uses today it is based on SPARC it did use spark sequel the insert overwrite in particular its implement that deletion its configuration driven so we are constantly you know their new tables are being added to our platform non-stop you just change configuration to do that Bacchus backups are supported so that we can for the first verify that a deletion was successful and we have a rollback strategy in place that we have to and also and I didn't talk too much about it but we've added some further feature support for relationships between tables I spoke a little bit about how you could delete our email address and also delete by other IDs internal IDs and so forth do translations between them which you know makes a lot easier to to do deletions across a wide variety tables it's scalable well because you use a spark and then a bonus we do have a repartition tool that comes with it and then we're currently working on open sourcing the project hopefully hopefully that will be out in the next few months so definitely can check the our github github.com slash gopro and it should be out there hopefully very very soon so thank you and I will I think we're in pretty good time home open it up for questions yeah definitely great presentation thank you how long did it take you to roll it out well the full development it would took us probably from initial concept so there were several revisions of it but the first version we were able to complete in about three months it was very manual intensive that version and you did have to do manual translations between IDs that you wanted to delete and then it was about three months for later so probably told about six months to the point that we're at now thank you that's so very impressive especially given that this configuration driven yeah I mean it's we've always attempted to to be as reusable as possible and we do have a small the data science and engineering team is pretty small at GoPro so we don't want to spend a lot of time we will take the time to invest in the infrastructure so the next time is a lot faster any other questions do you have a use case that you need to clean up animatic data for example a user spent $5 on product and you have an emetic data saying the sum of all the user purchased product is that when the GDP when the GDP are calm and not only you need to clean up the users data you also probably need to consider the analytic data as well right you're absolutely right and in this does also clean up so first of all it is a very generic table cleaner so any any any table that you have in hive or in SPARC you can clean up and yes they are most of our tables are the analytical tables so we have on transactional tables we have analytical tables which come from devices and from like software and then we have what we call aggregated tables on top of those and those grow very fast and so we with basically that's why it had to be configuration driven because after that first version which took about three months to develop we realized right away it was going to be a bit of a maintenance nightmare to have to maintain that and so yeah there's constantly new analytical tables are being added and that word or deleting rules all right cool thank you thanks [Music]