SBTB 2023: Sherin Thomas, Recipe for Building a Discoverable and Governed Data Platform.

SBTB 2023: Sherin Thomas, Recipe for Building a Discoverable and Governed Data Platform.

Recording: SBTB 2023: Sherin Thomas, Recipe for Building a Discoverable and Governed Data Platform.

hey folks thanks for sticking around to the end of the day I hope you are appropriately caffeinated ready for this talk awesome um so today um I'm going to talk about how you can think about making your data discoverable and governable in a data platform where there are so many different sources and types of data sets from raw to curated and so on and so forth and I'll be sharing some tips that I picked up in over the course of building this out and enabling Discovery in our um in our organization so my name is Sharon I'm currently a software engineer at chime we are a fintech fintech company based in San Francisco previously I spent some time at Netflix ly Twitter and Google um I am what they call a Sunday artist I love to paint in my free time can anyone tell me what they see in this image take a wild guess every Everything yes these this is these are like all the data tools and uh technologies that existed in 2012 fast forward a decade this is the number of tools and Technologies in the data space that existed in 2021 there are so many I literally cannot read the names of uh you know any of these the bottom line is data processing and data analytics and data curation has become super easy over the last decade but this has led to a different problem that of an abundance of data or too much data and because of this abundance finding trustable sourceof truth data sets has become really challenging good useful data is often hidden amongst duplicates inaccurate or ambiguous data and when you're looking for source of Truth ambiguity is bad I read this phrase somewhere when in doubt you can't be wrong and this stuck with me when you're looking for trustable data you just cannot afford to be wrong does anyone here know what Conway's law is have you heard of Conway's law at all um so Conway's law says that architectures uh often mirror communication structures in an organization so um basically a micros companies that adopt microservices architecture typically have like or you know separate siloed organizations and so on and so forth at a typical software company product engineering is a separate organization from data science or data analytics or anybody who's using data analysts and data scientists are consumers of data while product engineering are producers of data that's where you know they originate data through event events and logs and databases producers don't know what consumers need and consumers often don't know what producers have while the data engineer is stuck in the middle uh they are neither the producer nor consumer and they are trying to you know um bridge the gap and we need a way to make these groups talk to each other which brings me to the first tip of the day bring everybody to the same water cooler if you remember those old office pictures there are these water coolers at different corners of the office where people congregate exchange gossip you know like gossip what did you over the weekend you know that sort of stuff we need something like that in the digital world to bring everybody on the same page and exchange information and context and um and tribal knowledge when we set out to find something like this for our organization we had some requirements we wanted to onboard a data catalog block that can be used for many different types of users from PMs and engineers and Executives everybody and it needed to support a range of data artifacts tables and pipelines people roles everything and this data catalog also needed to surface the relationships between data how information travels through the organization so that you can get a full picture of all the connected component everything in a data platform moreover context is often you know it it lives in different places there is context in slack there are documents there are code comments there are um just you know in somebody's head and we need it to collect everything in one place and search and retrieve by relevance a place where producers can put context and consumers can find what they need and we decided to adopt adopt uh data Hub which is an open- Source product for um for it's a open source metadata platform that enables data Discovery data observability Federated governance it's um acral data is a company that provides a managed version but many companies are using an open source um uh the open source version of it and under the hood it's essentially a metadata graph it it is a graph of relationships between people and data and code and everything that you can think of and on that on top of that layer Discovery observability and governance uh can is applied this is what it kind of looks like so we when we go to the landing page we have data organized by domains by different um types data sets tables and so on and you can do search and by search by relevance you can you know once you find the data you find documentation about it full context about the data sets and lineage is my favorite feat feature especially the column level lineage we used to have so many issues where someone would like change the column type or um or you know like just add delete a column or so on and the the repercussions could be felt like many hops away and data this lineage gives us a way to run an impact analysis and make sure that producers are not breaking things uh under the hood the second tip is choose a common data language basically a platform agnostic way to codify a data model a schema definition language a Chim we use protocol buffer schemas pretty heavily and not just for serializing data on the wire or for serialization and deserialization we also use it as a representation of a data model even for files that are coming in even for to represents the data format of each row in a file that are ingested by an airflow dag per perhaps we also use it for defining service um you know like service apis and request response and so on and so forth but the more important thing is tip number three which is use schema for more than just schema and what does what do I mean by that so A very cool thing about declaring schema in code is all of this extra surface area that you get for free to enhance the capabilities of your schema to add documentation to add tags and so on and so forth in protocol buffer they introduced something called as message and field level options and these message and field level options can be used as annotations annotations to add extra metadata about a schema so here you can see we are using uh message level annotations to define a type of a profile event the team that it belongs to and we are also using the field level annotations to Define find uh classifications that I'll later discuss how we use it for governance and most importantly documentation adding code comments is the most natural way for engineers to add documentation so uh schema in code provides that surface area to do all of these cool stuff and the best part about these annotations and all of this extra metadata it's not really sent on wire so it doesn't increase the size of your payload on The Wire rather it's there in code for you to access if you need all any of this metadata and to me the most import uh interesting part about this is it opens up the possibility to use protocol buffer schemas as a declarative language Point number four keep the logical and the physical together now in any organization raw data usually originates at the ground floor or what I call the physical layer and and this data can be in the form of structured or unstructured events and logs and database entries change data capture um events unstructured documents so many different things usually it's not used as it is there is often some amount of ETL process like postprocessing a data engineer somebody you know curates it processes it using ETL workflows generates fact tables then generates curated data and it is often this curated data that is used for driving business ins sites for many different things mostly marketing analysts PMS or executives are using this business insights for funnel analysis many different things and they are far removed from uh the raw data or you know they uh yeah they they are really far removed from the many different layers that are involved in getting it the insid phas but they are not fully isolated from it if there if problems originate at the Raw layer at the physical layer or as I call it the it can manifest itself in terms of issues in in your insights for example if there is a data quality issue let's say there is a field that was just dropped which impacted your Revenue numbers it's very hard to see whether your Revenue numbers just dropped organically or was it a data issue and this is why it's important to have visibility into what's going on everywhere even at your physical layer to get a 360 Dee view to understand the impact of things keep the logical and the physical together bring Engineers PMS analysts business insights folks everybody to the same tool Engineers can add context add assertions data quality checks to the data they produce or curate and technical issues at the physical layer can be easily bubbled up to The Logical layers and this is where the uh the graph the lineage graph comes in handy we use the lineage graph to essentially bubble up any assertion failures and so on to show whether a dashboard might be impacted for example and visibility increases when everybody is communicating at the same place Point number five keep the logical and the physical separate I can see that I have already contradicted myself but please hear me out data governance is a gnarly problem rules of data retention and handling can vary so much based on where the data lives moreover a data can exist in many different forms as table as event in a stream in um in in a data Lake as a dashboard and the most challenging aspect of governance is applying rules and policies consistently across all the forms in which data exists also rules and policies may keep changing there is a famous saying um that all problems in computer science can be solved with one more level of indirection every data entity should have a logical model defined in a schema in a schema form in code something like protocol buffer and you can specify data classification or data governance rules on top of the schema using annotations and now rules and policies can be defined separately in a central glossery and Rule orchestration can happen at the platform level essentially three levels of interaction and this grants the flexibility and decentralizes the governance process let me give you an example data producers know their their data the best they know what they are sending they they are they are uh they they they know how to classify it but they shouldn't be concerned about who has access to what the governance aspect of it so what we have done we have empowered our users to our producers to tell us what the data means like is it a full name is it a SSN and just like add those classifications at the field level separately in by adding one more level of inter Direction uh another team the security and privacy team defines how this these classifications map to different you know whether uh full name is sensitive or confidential these These determinations Are Made separately also the rules of handling it should sensitive data be retained for more than 30 days or should it be mared these rules also exist separately in a glossery the m a mapping can be created between what Maps personal info to sensitive for example and policies can be mapped to rules and rules can even be broken down by platforms this way each platform can Implement their own Rule orchestrator and refer to the glossery for policy icies and the schema for the classification to determine how to handle data using this level of indirection we are separating The Logical that is the rules and policy definitions from the physical manifestation that is applying these policies at the platform and infrastructure level Point number three crowdsource metadata inje when we started working on this data Discovery and governance project the initially the team was the data infrastructure team was essentially like pulling data and we have many different systems we use airflow we have for stream processing we have kineses for batch processing we have spark we have a lot of custom declarative Frameworks built on top of these things even so and these all reside in different teams ml platform is a different team so many different things and initially we were pulling all of this context from these different teams and this wasn't scaling and we were moreover we were trying to understand what what context is useful what is the metadata that we should be collecting but data platform or infrastructure we are neither The Producers nor consumers of data we are just the people providing the shovel or the pipelines to streamline things we deployed something uh like a squad model so instead of doing it within the data infrastructure team we put together like a squad or a group of data stewards with representatives from different teams so now we have representation from ml platform from bi security uh data engineering and each member in the squad brings special skill sets and perspectives as to how what metadata needs to be collected how must governance rules be applied for example and since then we have been able to ramp up our impact metadata inje should not be data platform's responsibility alone and design the other principle um that we really went by is to design metadata ingestion to be as close to the source as possible so we went from a pull-based model where data infrastructure was pulling context to a push-based model think log and Metric collection even metadata needs to be collected at The Source on an ongoing basis our Squad built a central metadata collection API which is a wrapper over data hub's own apis and now with this team we are building metadata extractors which essentially convert the um the uh the source model into a metadata model that is understood by data Hub now these metadata extractors are crowd Source they live in the platform that is supplying the metadata extractor and it is continuously pushing this metadata to data Hub and uh we we don't have to do anything it just continuously uh being pushed and kept up to date Point number 7 everything must have an owner and this is an easy one but still very very hard to get right no owners no accountability and as you are thinking about cataloging data sets make sure to assign ownership from the get-go also consider different types of ownerships a technical owner might be the producer or curator of a data set and responsible for managing incidents and so on a business own owner might be responsible for appropriately cataloging contextualizing and asserting on key slas for the data set point number eight meet data consumers where they are one thing we noticed was um it's very hard for people to hop from one tool to another we already use so many tools we have slack we have Google Docs notion so many different things and if users have to hop from one tool to another to find context inertia sets in and you know they end up just going to slack to find answers rather we wanted to bring context to the consumers to help with that we use a chrome plug-in which was uh built by data Hub the acrel the acrel who um manages data hub for us they this plug-in essentially we currently have a plugin for Looker and this brings all the context about a data set that a bi person is looking at into their own tool so now they don't have to go to a separate tool everything is available in a chrome plug-in tip number nine shift left shift left is a generic term used across the industry to Define an approach where responsibilities are shifted to earlier in a software development life cycle this is commonly heard in reference to software testing where quality assurances moved to earlier in the pipeline we wanted to use a similar approach by moving cataloging responsibilities to earlier in a data life cycle which means giving cataloging responsibility to producers I don't know if you realize this but now but I'm a big fan of schemas and schema annotations and as I mentioned earlier in the governance example at CH we use protocol buffer schemas and schema annotations pretty heavily and schemas are a gold mine of information we have context in comments we have um we have classifications we have a lot of ways people you people are using annotations to catalog what we did was we used um protocol buffer SDK we built a protocol buffer SDK to essentially ingest all of these comments scrape comments from schemas and push that as documentation to data Hub also the annotations are converted into tags or glossies or domains that adds like special indexing on top of this information to make search and um organization easy with this ingested into Data Hub now we can search across the entire ecosystem to find what we need moreover we can slice and dies by tags we can organize by tags and so on I save the last best one for the last which is data contracts this is a hot topic that's been at least it's been showing up very frequently on my LinkedIn and it has been you know there's there's been a lot of conversation happening around data contracts over the last few months but what is data contracts to explain that first let me bring you back to the first slide where we established why Discovery and governance is hard the root of most of our problems is a disconnect between producers and consumers data contract adds a formal agreement between producers and consumers where one didn't exist before think slas in this world consumers add assertions and producers own accountability for any assertion failures throughout this presentation we already talked about a few key components that when put together can form a data contract first is schema and this is the bread and butter for any SLA a basic level of validation of the data model is required but that is not sufficient you also need semantic validation this is where data quality checks come into play there are many data quality tools today like Great Expectations um anomalo many many different things we are using the one um within data Hub we have also built our own custom um data quality checks using airflow Dags and that live very close to the transformation logic but we are surfacing it via data Hub this way users whenever they're looking at a data set they know whether there were semantic failures whether there were any like data quality check failures but whatever tool you pick think about how you can streamline assertions and uh contract creation consider a surface for declaring schema and semantic assertions the other component of contract is account accountability and lineage is an really important tool for accountability it gives producers a view of how data is used Downstream so now you can hook it up to any schema changes to check whether you know it is any kind of schema changes it impacting any Downstream reports or usage and this Bridges the gap between producers and consumers that didn't really exist before similarly consumers can get notified for any Upstream changes that impacts them so there you have it folks um all 10 points uh to recap the 10 tips that I went through today um bring everybody to the one water cooler choose a common platform agnostic schema language for everything even for files row think everything in the data world and use schema for more than just schema keep the logical and physical together but also keep them separate crowdsource metadata inje everything must have an owner from the beginning and meet consumers where they are shift left give producers the ability to Define what they are sending and embrace data contracts and that's it thank you so much thank you Sharon that's very good yeah thank you and you are right on time actually we still have five minutes or so so feel free to ask questions uh to Sharon uh anyone of have questions so as okay so as companies are starting to introduce llms into their systems what are the places that you uh Envision uh large Enterprises who probably already don't do data governance very well what do you predict they will fail at as they go to llms and very data intensive new structures yeah so I so my hunch is at least in this area for structured data it is still you know it is easier to know the nature of the data or you know you can rely on people to tell you via schemas and so on and so forth it becomes even more challenging for unstructured data do you you know like where is the data where is the source of data what is the data quality what you know does it contain any private information and so on and so forth so what I foresee happening and which I'm already seeing are happening quite a lot is a lot of like data quality checks and assertions and contracts are becoming more important ways to essentially find the nature about the data whether you know does it contain private information and so on and so forth and also the provence of it where is this information coming from um is it you know appropriately organized by that like is the context where the data originated and so on so this is I feel like we with llms and all we are going more into the unstructured data world where um where you know there are different completely different set of challenging problems follow but when we look at Vector databases that are essentially unknowable what what does what do those concepts mean with a black box like a vector database yeah that's a great question so even with Vector databases I think it's very in my opinion it's very hard to do it at a very like granular way you know for example it's very hard to know whether um some an entry in there like what is the nature of that so my guess is and I I I haven't really worked a lot with Vector databases the granularity of where governance happens might happen you know at the table level there may be certain data that you may just not store or make it available for retrieval or you know in in your um um yeah in in the data set so I feel like that's where it might just end up happening yeah thank you yeah anyone else have questions for Sharon yeah no no okay yeah I guess you can always still go up to the Q&A Island up on the fourth floor to uh talk with Sharin to afterwards right if you're still around to yeah okay great awesome all right thank you thank you yeah