Scale By The Bay 2019: Thomas Gerber, From datasets to tables in a multitenant data lake

Scale By The Bay 2019: Thomas Gerber, From datasets to tables in a multitenant data lake

Recording: Scale By The Bay 2019: Thomas Gerber, From datasets to tables in a multitenant data lake

[Music] hello everyone and let me specify that very quickly plan migration we're still working on it but yeah my name is Thomas I work at Salesforce more specifically I work on the i9 platform that is the it's AI layer of Salesforce and more specifically my team owns Einstein derelict that feeds data the provides data to ml systems throughout source for modeling or scoring purposes and so today I'm gonna tell you why we are building a new abstraction in a very lake called tables and why we thought that what we have currently datasets is not enough and I hope that so rational I'm gonna lay out for you might be useful in your own companies for your own Lakes so let me start by the end the key takeaways are as follows Amida bill datasets create coupling between data consumers and data producers especially for firehose kind of the producers meaning you have an object that you care about in your lake and you new records or updated records come in for the project on a regular basis and so that coupling makes life harder for consumers because they need to be aware of how the producers puts data in the lake instead we think that adding a new attraction called table think of it by now as virtual spreadsheet if you will happened on a spreadsheet it essentially hides the hydration from the consumer the consumer simply knows that there is a table an object is not anymore ever-growing set of data sets it's just this ever-growing table and we think that not only will that make life simpler for the consumer just on the face of it because it doesn't have to deal with a set of data sets but this table abstraction will also enable us meaning the team owning data like to provide some things that we cannot do today on behalf of the consumer that is schema evolution hey what is my object now one of its field was renamed versioning or time travel being able to see the data as it was in a point in time and better predicates pushed down so that we minimize what's being sent to the computer used by those ml systems today unfortunately it's up to the consumer to deal with that and this is usually teachers or tricky so in essence what I'm saying is data sets were very nice are very nice for their producers you can just keep adding data sets to your legs that's great but it makes life hard on all the consumers of your lake instead with tables it's a bit reversed it's a bit tougher on the producer but it's much much easier for all the consumers of your lake and since we have many more consumers in south course of our lakes and we have producers we think it's a good trade-off to make we make it simple for the menu so those are the key takeaways keep them in mind as we go through our journey so let's start by some context so as I said I work on the eye-shine platform which is an AI a layer at Salesforce and so if you ask a data scientist on platform hey what is the main lesson learned he will tell you he or she will tell you three things customer specific models beat global models that means that as sales force if you have a feature that try to predict say churn on accounts it's probably better and I will explain wine later to create models per customer so be more accurate than a global model that applies to all customers running is number two the many many use case at Salesforce that can leverage ml or AI and way to view data scientists two points together mean to automate the creation and the operation of a lot of ml pipelines at Salesforce essentially at least one per feature per customer and Bob that's why that's why we hear the silver lining though which is the third learning is that the maturity of business data that we do is is structured and this is boon for the data scientists because they can leverage this structure to automate machine learning as much as possible if you're interested this is not the purpose of this start but if you're interested check out our open source o to ml library quando called transmogrify it's pretty cool I'll link a slide at the end now let me double-click a bit why customer specific models be clubbin model and what is the difficulty there for those who are familiar with salesforce account is one of the most standard object it's what sells rep around the world on a daily on an hourly basis interact with resources UI to track who they're selling their product to and so it has attributes like name address and a lot more that i cannot display here the power softwares however is in its customization capabilities if you work for Acme Corp you might see as a cell rep you might see a custom field or a set of custom field in there a specific to Acme and that have been added to the schema of that account by a sauce was admin attack me in this case for example train risk that would be different if you work at Wayne Enterprise and I have the whole set of custom fields and if you work for Weyland you might have even custom objects that have been created by the admin and so that means that in our lake we have to deal with a lot more that you might have thought a lot more schemas that he might have thoughts tens of thousands because customers customize objects they work with so that's obviously create some difficulty for us but it's very important that we do capture those custom fields those custom field have have been purposefully created by the customer and therefore they are very likely to contain information that is predictive in nature and so we would like to love ml system that sauce was want to leverage that information so it's very important for us to grab that and unfortunately with that comes a whole slew of difficulties like handling schema changes and so forth so as I said my team owns the data like that provides data to those ml systems so a data Lake is usually a simple compliant scalable data API something specific to Salesforce is that we had to work very hard to ensure tenant isolation meaning like an ml system building a model for when Enterprise should not by accident use data from acne that's very important at Salesforce and so we had to do a lot of work to make that a reality and our data Lake is optimized for ml access patterns meaning get all the data for modeling or get the latest data for micro batch predictions it's not a general-purpose like it's an ml Lake and one of the things that we had to work a lot on is automate the hydration of that cake meaning like for every model customer model that is being built at Salesforce we need to essentially automate the constant hydration of the lake for that object meaning for example account at Acme and so when we started we decided to go with the simplest thing which was immutable data sets when we're gonna see an example in a few seconds it was the simplest to get started and it was also the one we felt would scale well at the beginning and so what is a data set is essentially metadata about some piece of data that metadata leaves in a metadata database and contains pointers or locations where the actual data leaves in a blob store pretty straightforward let's take an example this is an example of a data set and so let's start by the bottom data sets usually contains attributes that are used by the system that's going to consume the data say it's a spark or tensorflow or whatnot that needs to get the data to build a model say and so those fields are usually EEMA and manifest list that contains a location of the actual data and maybe some stats that are that the framework we use can can make good use off they also usually contain metadata related to the lifecycle maybe a status some lineage this is what you would use to do GDP our kind of processes and compliance in general and then it contains attributes that will help consumer system discover the set of data sets that they need for the task at hand some obvious one is hey what is the object you're looking for and so that for us is the combination of and this example account and tenant meeting customer in this case Acme and then what I could generally describe as tags for example the type is it a snapshot or an incremental data set and a few other things that will help the consumer system query the data lake and find the right data set for the task now you might already be able to tell that unfortunately those tags essentially leak hydration details to to the consumer and the consumer of the data lake needs to understand what those tags ours are contract between the producer and the consumer and they need to be aware of this in order to properly request data and what's interesting is the consumer system knows what it wants in terms of the data we're going to see a few examples but this the the consumer to translate this data request into a metadata request through those tags or sue those attributes could be first first class attributes doesn't really matter but they need to translate that in terms of metadata so they need to be aware of what those are that's all bit unfortunate because consumer system really don't care so just want a schema some file locations and we are forcing them this is attraction to be aware of that abstraction so that's already I would have one limitation of datasets makes life already more complicated from the consumers they need to be aware of that another set of issues is that through the abstraction me as the owner of the my team as owner of the data Lake cannot provide more cool things to the consumer for example the handling of schema drift as we mentioned earlier it's very important for us to capture all those custom fields that change over time and today we cannot do a great job at it let me explain why here is a timeline of data sets that was created just for the sake of this example and so at time 3 here a snapshot was created with thousand and one records the schema at that time was ABC very straightforward at time for smaller data set was created with just four new records for his say account at acne but something major happened at that time an admin change of fuel type rename the field B into Z and that time v lemon change again the schema and removed field a say in also is a very small number of new records for the object as an as a consumer system you just want to ask hey get me all the data for that account for Acme and that way I can build a model that's ideally what you want unfortunately assuming you you work your way through the metadata you will probably get two dataset three four and five but you will not have the knowledge that field Z is simply a rename or field B you either need to as a consumer of the day like figure that out by yourself error-prone or you need to accept that you might not be loading you might not be loading the values that were in field B for dataset 3 which is unfortunate again because so much of our accuracy in our automated machine learning depends on getting those custom fields what you would like ideally is simply not have to care about that and just have the data being aligned for you another limitation reproducible discovery you can ask say at time 4 you asked data like hey get me everything that's new since time 3 again you can probably work your way through that using those metadata you will get data set for that's nice you do the exact same request at time 5 you will get two data sets you have no way today to say hey I want everything that's new since t3 as of time for and that is a problem for ml systems because they for troubleshooting and for consistency they usually want to be able to feed again the exact same data into one of those processes to again help with troubleshooting or help with consistency and so today we don't provide that as part of the data rake and the customer has to deal with that mostly by tracking data set IDs and finally the third thing that we cannot get to do with that abstraction on behalf of the customer is better pretty good push down I actually understated something at the being I said customer models beat global models that's an understatement in the sense that segmented customer models now beat customer models meaning like if Celsius wants to churn at acne it should not necessarily create 100 but maybe a set of models one purged graphical region for example and so ml systems want to ask want to now have access to just a segment of the data for example get me everything since t2 where geo equals us now this is interesting because if in that tag system or whatever attributes geo is not surfaced tough luck you cannot request just that segment to the current API and you will have to get again three four five all the data and deal with that on your end as a consumer system that's unfortunate what you would like it is to be able to say hey I just wanted us geo and you get a subset of the data directly so all those three things we cannot easily do when we have 1/2 of the consumer of a derelict and they have to do it themselves this is so those three reasons plus the fact that the consumer cannot express its data request directly has to go through a metadata layer pushed us to investigate a new abstraction tables so what is a table exactly as I said earlier instead of thinking as a four for an object to be a collection of a set of data sets and every boring set of data sets think of it as an ever-growing table in our case a table an object is defined by an entity say account for a given customer say Acme and you have a table ID now as a consumer system you can request data by posting what's called a scan for the table ID the canonical scan is empty scan which is essentially asking hey what do you have in your table in this case the end response will be okay schema and manifest mostly that's what you care about as a consumer system of the lake and so you will see fields E and C and you will see a bunch of paths that properly align with regard to schema drift if you want the latest data you just add a pushdown filter for example x time greater than three you will see almost the same scan response except the manifest will be much smaller and so your system can will consume less data trust the data it needs ideally that API should also support time travel so through a primary from example as of time for you should see two differences here you still see field a because it was not deleted at time four so it's a Z and C and you don't see the latest blobs of data because they were not there as of time for finally get a segment of the data well it's exactly the same for latest data you add a pushdown filter geo equals US doesn't really matter what your field is the data leg should attempt to prune the manifests as much as possible and turn something that's as close as possible to your filter so this is ideally what we would like this abstraction to be now as I said earlier we're still in the process of building cz's and we are currently trying out the implementation of that using iceberg which is a project that came out of Netflix and now an Apache project an iceberg is a table format that makes a couple of important things for us easy or simple first it supports scheme evolution with no side effects so it's simple to add/drop update or rename columns correctly it to the format it's also easy to do time trouble which again we said what's important for ml systems and iceberg allows for partition layout evolution which means based on the daily volume and the query pattern we can address the partitioning so that Prickett push downs are more effective when a request is made to the lake so again this was just painting is a rationale for us to work on on this abstraction and again the main idea is data sets were great for producers you just push data sets push this let's push this that's very very straightforward unfortunately you make your life a lot harder for your consumers because I need to be aware of your metadata scheme essentially and they need to be able to translate the data request into a metadata request and then they had to do all the difficult stuff around schema handling a pretty good push down and reproducible queries on their own we stables we're gonna make it harder on the producer of data but consumers will reap great benefits and because I'm many ml systems at Salesforce that want to consume data we think it's the right thing to do thank you very much for your attention I'm gonna take some questions just before that and just showing you a slide about transfer or grafted AI if you're interested in how we automate machine learning take a look it's pretty impressive and it if you have structured data it creates a pretty good model interpretable model which is important in those days for you to use so thank you very much [Applause] I'm 12 it helps if you repeat the question up there I'd like to thank you so I have a question regarding any messaging platform that is used by anything so I'm not sure I cannot hear it super well a message platform yes the idea behind this question is that if you integrate with systems like this they already allow you to take care of schema validation and scheme evolution that does exactly what you were showing in the first place yes so it's a very good question my answer has so the question was do we integrate with a messaging platform because or subsystem because that could take care of schema drift for you the answer is twofold I think Salesforce is a collection of systems now so you might not have that for every data source so that's one answer the second answer is the second part of the answer is those are still fairly new at Salesforce even those who have change that I capture this is still a little bit new and so we have been relying a lot on pooling and slowly moving towards stream but I would I would also add though that tables are much better abstraction when you when you are ready to get stream data into your data Lake because you don't have because the data set doesn't make a lot of sense it makes a lot of sense when you do pulling it does a lot less sense when you do stream hydration of your so even in that case I would still have use that table the better abstraction hi sorry bit loud I guess my mental model for what you're talking about is essentially a table is like an apparently mutable virtualization of underlying immutable data sets that are sort of nominally supposed to be continuous but they can have different schemas and metadata through time right that make sense yes yeah it's correct fantastic thank you and and it's that's exactly right you hide what's underneath so that the consumer doesn't have to deal with that I deal with that on their behalf thank you very much [Applause] [Music]