SBTB 2023: Fabiana Clemente & Gonçalo Martins Ribeiro, "Unlocking the Power of Data-Centric AI.
hello everyone so welcome to this last session before lunch um and well today I bring you unlocking the power of data Centric AI while mastering data preparation for machine learning my name is Fabiana Clement and I'm Chief data officer at Y data and well just before bringing you all the stuff around data preparation um a few words about myself um my background is in applied maths and data science I've worked from Big Enterprises to small startup so I do have my fair share of differences between a data architecture in a big Corporation and really how do you operate data at a small scale well not a small scale but at a small company for for sure um and well my passion definitely is data science though um uh I have a particular interest for time series and generative models um feel free to connect with me on LinkedIn more than uh happy to exchange some ideas especially after the talk or any other day um and well to start with and why I'm going to start by foundational models well data preparation have been for quite some time around right but especially in this era where we see foundational models everywhere uh from images to text we have Dolly we have chat GPD they have um changed the the perspective that we have around data and how we prepare it this models these foundational models they are trained on Broad data sets of unlabeled data which makes makes them of course Very Special they are um as many may may say probably the future of AI but and they are not only reusable but they are also very flexible when we compare them to previous models of AI which makes them quite applicable to any domain or any industry but even though these large models can cost Millions sometimes to train um each New Generation Um produce more and more accurate results and this is very enticing for the future of the the industry and well we might even think that given that these models trained with a lot of data are so flexible and so good this means that we might not need uh great quality data after all we can train in big pools of data and that will ensure us results anyways and Truth to be told we have seen the impact of these models as well on the landscape of tooling as you might imagine so the impact is not only at the application Level but also on the tooling of AI so large language models are definitely bringing new tooling to the space new ways of developing Ai and all of that is kind of reshaping how we see the AI development but one thing haven't changed yet or at least that much you might think garbage in garbage out that's over no more problems with data we got it sorted not quite so we are still having the same issues as we used to have on the data preparation side so on on the data preparation on the quality of the data that you have so some might say it's the labels that are inconsistent some may say that is the data that is not enough for the use case that you want to solve is the representativity of your population there are a lot of aspects around data that still impact this large language models and you can take for example articles like uh we have seen published like chinchilla where the a model that is way smaller perform way better than a model that is bigger trained with more data just because the data that was used was accurate and well that's why and still even though we have foundational models in the end it is still all about the data so far um the the if we go and check the available open source foundational models and that was actually referred in many of the talks today um they are still trained on open- Source available data which means that in order for an organization to differentiate or um even someone that is just trying to use foundational models in an application they will need to have special attention or pay special attention to the data that they use either to fine tune for the rag or even just to train again the model and that will be the difference between what you see open sourced and the performance that you will have from training your own model and well that brings us to the data Centric AI aspect of AI development so far our Focus have been on the model so perhaps many of you know um this process by doing a data preparation as an engineer you kind of clean some masses regardless of the model that you are going to use you then have a model that you train once and you do hyper parameter tuning at this point when you are doing your hyper perimeter tuning on the model your data is fixed you assume that that's your artifact and that's the data that you are going to use uh for your model regardless so the only thing that will change and iterate is actually your model parameters in the data Centric context this is a bit different what is said and this was uh uh even stated um by Andrew Yang so he was the first to coin the term data Centric Ki is that this process shouldn't be like this so in a nutshell your data preparation shouldn't be completely detached of your end application so if is a machine learning model or even an analytics application you need to optimize your process or your data preparation towards your end objective in this case your fixed system is the code let's say the model and not the data and the data is actually your iterative process so it's like doing hyper parameter tuning of your data preparation of even your data that's what I call treating your data as your product so thinking data as a product means that you should put your data um in the context of the business needs and at the heart of your data flows and different data preparation design so don't think about your data preparation or your um ETL job as something that is independent from the rest it's actually part of your product it's actually part of your development uh and it's a living system that needs to be iterated so a question that some of you might have but what does it mean to have this high quality data to achieve the highest results and what does it mean to iterate this data what is the objective what will we focus on so it does depend does depend on your goal it does depend on your use case and actually data quality and what we call um to be that the objective of a great data preparation for AI have multiple faces and will depend on the context and use case so just for you to have an idea there's an article that states that there are more than 70 aspects that can be measured as data quality so it can be just accuracy completeness consistency Integrity it does depend but those aspects are the ones that you should add to your objective function when you are trying to optimize your data preparation flow and well this is very good this is very interesting but how can we just translate data Centric AI into practice what are the best practices that we can follow and what are the tools that we actually can adopt on a day-to-day and while doing our data preparation in our process in order to have better AI development one of them perhaps for many of you it's obvious or it's something that you already do on your dayto day data profiling so in a nutshell data profiling summarizes all the essential properties of a data set allowing different data sources comparison and uh a thorough analysis of the available data better data management means improving your data quality over time what do I mean by this many uh in some organizations or in some places and we actually have seen that in the uh previous talk this step is then manually and in an adoc manner many people does build their own validations every time they need to check the data they build their own visualizations every time they need to check the data and the fact that different people end up building different systems to validate the data it leads to a different understanding of exactly the same data this might seem like something surprising but it's not that uncommon if you do the same different validations or you have a different understanding of the same question you will Implement validations that are different and that will lead to different results so that's why the data profiling should be in a nutshell a standardized method can be defined by the business it can be standardized based on the machine learning model that you want to use or even can be just um based on the the the end system that you want to build like a bunch of expectations for example that you want your data to meet but what is important at that this process is standardized since the beginning and is actually the first step in your um in your St uh data preparation flow and have in mind this step you won't be using just once you probably will be using several times throughout the process of the development after all you want to validate always the changes that you are making the decisions how the decisions that you are doing in terms of the data preparation might change or are changing your distributions or the data behavior and of course data profiling can be done uh with several different uh components it can be done through visualization a lot of teams do prefer visualization because they are actually more actionable when compared to other methods can be automated analysis through statistical methods and scores or it can be just by expectations in business rules it does depend on the use case it does depend on your end goal second and this is probably uh one that might be not so common uh but definitely one that is dearest to my heart is synthetic data so synthetic data is something that uh in the toolkit of data preparation can make a lot of difference I'll start by the question that you might have at this moment with is which which is which is what is synthetic data synthetic data is artificial generated data that was not collected from Real World events it keeps the same characteristics as the real data and is not traceable back to original individuals so how can one use synthetic data just to make better data preparation so it might help you to have access to data way faster because if privacy regulations are in place or you have a tight security system you will H want to have access to a data that has no privacy concerns but still holds the same value as the original one and that's what synthetic data can uh deliver you or perhaps you might want just to stress test your system during data preparation and you might want to have a data set that actually reflects a corner cases edge cases or different types of behaviors based on your understanding of the business sometimes the data is just hard to collect you don't have those events but you know they might happen so synthetic data can be an answer and well there are different types of synthetic data and not all the synthetic data is the same probably some of you are already familiar with what is called Demi data so this randomly generated data that holds uh the same format or the same looking fill as the real data but has no value to be used for analytics or machine learning so it's just a dumi data set it is great to test schemas it is great to test data flows but it's not that useful if you want to build something else on top then you have what do we call datadriven synthetic data and this is where we use gener ative models to generate synthetic data that holds the same value as a real one but have no privacy concern so in this case we use models to learn the original distributions and make those distributions replicable so it's not only a case of you can generate more data but actually you are compacting your business information into a single model and use it every time you want new data to share with different teams for example to build um sandbox systems or just to test different scenarios and that's where simulat data comes in so this is the case where you have currently for example a population uh and you want for for example for a specific City you have data from a specific City and now your organization wants to go to cdb they have never sold anything on cdb but they want to be able for their system to still work when they start their business on CDP so what you do is grab your um your data from the first city or the Cs where you are actually or where you have data from and you can simulate the data for cdb under assumptions that you know from the market that's where simulated data can be useful this will allow you to well not having problems or not having data and starting right away to operate on a new location will allow you to have more robust systems and more generalizable systems as well so very quickly why go synthetic it's cost efficient when compared to data collection so if you have no data and you are able to simulate it and have it with uh the right accuracy that you need for your use case why not you cannot unlock data sharing supports responsible AI after all you can use synthetic data to balance uh populations for example there are under represented it can bring more machine learning performance it can enable faster product testing and development but of course and especially around data preparation and this was also mention in the the last talk nothing is more important than reproducibility and when you are in a data Centric AI perspective and you want to make sure that your whole flow of data Centric AI um is aligned with your objectives whether is uh correctly measured and you can make it reproducible for the time or even versionable data orchestration is a very interesting option for example in let's assume data pipelines you probably are somewhat familiar already with the concept airflow dogs all of those can be considered data orchestrators you can use them and they can be uh very essential for organizing tracking versions and ensuring the consistency around your old data preparation they manage the flow of your data they clean they structure it uh the analysis and they can also allow you to integrate very effectively your data preparation along with AI and that's very important especially if you are optimizing your data preparation towards for example a machine learning model by incorporating versioning and reproducible features um a pipelines can enhance the reliability and transparency ensuring that your data processes can be easily traced and replicated so for example let's say and this is a very common decision you decide to you have a specific data set and you decide to standardize a specific distribution we all know that in case the distributions are skewed this is the best uh might be one of a good option if you want to build a model on top of this data you run a certain decision around standardization and you check how does that impact for example the scores of your model afterwards and because you know that you were um trying two different uh standardization models you want to test another method and you want to compare the results of both in order to decide the best data pipelines are the ones that allow you to do that version and allow you to make that Pro possible process not only comparable but also versionable so you won't lose the first decision the decision of the the the model A versus the model B you will still be able to decide between a specific standardization or another one another example might be for example how do you deal with the consistent records so records that have issues do you delete them do you uh impute them or do you change them all those decisions do impact your machine learning model in different ways I'm not saying that one is worse than the other it just depends on your business on which one might be the best and of of course on the issue but you want to be able to test the three of them and choose the best one or the one that optimizes the best the performance of your model and that's where pipelines can be really be helpful but of course all of this is super important but another point that you should always take into consideration and along with pipelines is data Version Control so it doesn't matter you have a pipeline you have the code of your data preparation versioned you have your code for data preparation comparable and traceable but you don't have your data version that's probably uh the last element that is quite important for your to making your proc process fully reproducible um and robust you can use of course GitHub is a possibility but definitely not the best practice but there are other tools um specific for data Version Control that you can use you can of course set up your own it depends on the organization but that that is a must when to combine with data orchestration otherwise definitely even though you have your uh all flow fixed your results won't be reproducible okay so with this I want to end our my my talk I want to enjoy um invite you to join uh fabric Community version this is the product that Y data is developing we have a data catalog with automated data quality profiling in case you want to make sure that you standardize your analysis of course we can help you with any synthetic data generation that you might require and last but not least all these processes can be versioned and integrated on our data pipe data preparation flows uh and in case you just want to give it a try experiment and you really like code we also have open source uh both the profiling and the white data synthetic so you can build your own um your own systems your own flows and try it yourself perhaps integrated with uh with your tools or just locally I hope you have enjoyed and thank you I think we have some time for questions so feel free okay so thank you everyone oh hey there iting youbing on that's that's a very good question so usually and this is uh well the advice I gave here we usually use also for the for example the synthetic data generation process uh the best way one thing that highly impacts exactly the accuracy or the performance of how you can replicate very well the real data is the model that you choose for your data so so depending on the metadata and the profiling of your data different models will play best or have a best uh uh performance so for example if your data set is very small and it have a lot of categories you might prefer a generative model like AIA Network that is smaller but can understand a bit better uh the causality between the different categories but let's say that you are dealing with a large volume of data very big um um with a lot of numerical variables and just a few categories so scale is important for sure but uh and you have um just a few categories and a lot of numerical so you want to have something that can assess multivariately very well your data you can go for a transformer for example uh or even again they will uh be able to replicate the details of the distribution of your data of course you might might be want to pay attention where you have imbalanced behaviors as well so in that case the G is not the best and you might want to go uh with a chained version for example of the G which allow you to condition the generation for so you can have that replication of the less represented uh um population let's say so it does depend but usually if you know really well your data the generative model that you select will ensure that replication but you really have to know the data okay I'll be available for Q&A uh later on so be happy to chat with you um and thank you everyone