SBTB 2023: Guang-Jie Ren, Data management for enterprise-grade foundation models.

SBTB 2023: Guang-Jie Ren, Data management for enterprise-grade foundation models.

Recording: SBTB 2023: Guang-Jie Ren, Data management for enterprise-grade foundation models.

um hello everyone my name is uh Guan ran I'm a research scientist at IBM research uh we actually have a research lab in uh South San Jose called the almadan lab and um um today I will briefly talk about uh Foundation model um or in other terms larger language models generative AI um as a background or context for the data management work that we are doing in IBM and I will dive into some of the details um for us to discuss in terms of challenges and opportunities as well so in terms of uh Foundation models um the way you can look at it from a historical point of view is um really it started off right uh in 1950s and 60s as exper systems and then got into um the era of um machine learning um that's when we start to have uh quite a bit of data and starting from the 2000s um having a lot of data because of um social media and uh Internet of Things um but still very much uh a lot of data in order to build models uh you have to manually label those and that's quite expensive um and then you need quite a bit of compute if you go down the path of doing a lot of deep learning so deep learning has uh come along um in the past 10 years also to uh specifically look at uh not just language but also uh image and so forth um but what happened the last 5 years uh by introducing Transformer as the architecture for deep learning uh you start to move away from labeled data to um uh unlabel the data uh sell for supervision um of course with the uh uh ability to have more and more compute nowadays um that leads to the uh emergence of foundation model and the way we look at it from IBM point of view when we um approach a foundation model model is really three parts the first is the architecture as I mentioned so it can be encode only um and it can be encode decoder combination right um but what's most popular is really uh the decoder only architecture uh that powering gbt and other generative AI models the second component which I will dive more into is really the data component and then the third is the uh uh compute now um if you haven't noticed uh what happens um since the release of chat gbt um it's really a a new arms race in terms of how bigger you can get to uh when it comes to larger language models and uh there are hundreds of billions if not trillions of parameters available for for the uh Transformer architecture um and you are able to put in a lot of um context as input or prompt into these models now uh in fact a single um novel can be now um provided as a prompt into one of these models and if you look at the data actually there are a lot of um uh tokens being consumed to train the models and the largest one is as you can see gbd4 13 trillion tokens uh that have been uh prepared and fed into uh training uh to prepare the U the model to have the uh language capabilities now IBM takes a slightly different view of course we think the size is very important that's a u um a um fundamental uh feature of uh larger language models uh but also because we deal with Enterprise clients uh we think that maybe uh suitable size model but with better uh governance and a better Mitigation Of risks that come with these Foundation models is also important um we actually released our first series of models called Granite models back in September and one of the things that uh we um put out in the Market is we actually are able to indify our models as if it's a piece of software um as we usually do right to our commercial software um and I'm I'm going to dive into the reasons that we are able to uh have the confidence uh in doing that now in order to get to the so-called Enterprise gr uh Foundation models the very first thing that we have done is to look at the training data set and try and include as um much uh industry and domain data as possible um as you can see we actually include quite a bit of uh financial data in our training but also legal and other um uh typical domain data that's potentially left out uh from other data sets so that's something that we look at from both use case and uh data set point of view now unavoidably in order to do the training you have to uh include uh a lot of So-Cal internet data and uh right now there are a number of sources of those but one of the most uh famous source is a called a common craw um and that's a snapshot of the internet every month right and uh it goes back to uh I think 2012 also Al 2008 also so you have um 90 also snapshots around uh to be used as a um data source for training now there are a lot of problems of um uh this data from the internet as you would expect and um and that's uh in part in large part causing a lot of problems uh for the model um purely from data quality and uh um risk point of view uh there are quite a few issues um there are obviously inappropriate uh sites and contents within right and there are span sites and they can be duplicates of legitimate sites so there a lot of duplicates um even in some of the um legitimate sites there are a lot of uh low quality content junk content uh that you don't really want to kind of get into the model in the first place um and then uh the data itself has a lot of offensive and toxic uh content um and a lot of personal information as well U from uh demographic and the uh representation point of view uh they are pretty biased right um because of the internet users and how they see their world and represent the world as is um and the funny of given the emergency of um Foundation models there are a lot of machine generated data in the internet now uh so you start to form almost like infinite Loop of uh uh AI generating some output and then Al those output gets into the model as well and last but not least um um researchers have found a lot of benchmarking data sets in the crow the data so essentially you are train training a model right on um supposedly um objective uh evaluation data sets and of course you kind of score quite high right in certain data sets as well so that's the problem that we are facing and in order to mitigate these uh risks um uh we have um uh come up with a number of approaches and this is something I want to share and then uh hopefully um have some discussion with you all um the first is that um very quickly um IBM um is known for its a Enterprise brand right and we kind of take a risk and uh exposure uh very seriously so we put together a couple of uh teams to specifically look at what data we are taking in and can we look at them from technical business and legal point of view uh if and how we should take them in um and then when we do acquire the data there are multiple ways of acquiring those and they're governed by laws and regulations in the US in Europe and in um other countries differently so how do we uh go about those um acquisition uh decisions and that's important as well now equally important we have come up with a pipeline and this is um uh in a way typical to what you would do when you have data for machine learning or AI project uh you want to pre-process those but given the largest scale of the data and the variety and the quality issue that we are coming across actually this is becoming critical to our mission uh even before you train the model um so as you can see on the uh uh flow chart the corus itself as they come in is in pites uh scale um before the text is extracted um and there's a lot of uh um binary files and other content that we don't deal with yet right for uh language model so we do the text extraction as the first step and then as I mentioned earlier there are a lot of duplicates and and this is where it becomes interesting so some of the duplicates are exact D duplicates and some of the duplicates actually U by asso iation by similarity so we do a large scale uh D duplication um and as I mentioned earlier so if you look at the web data that we deal with uh those are probably uh trillion um uh in size for each of the snapshot and if we want to do D duplication we have to do across dat across 30 or 40 snapshots um so that's a very large scale data operations that we are running and and then in the middle as you can see these are the annotators um we do um annotation on the language itself whether it's English French Spanish or otherwise um we do the uh uh document quality annotation um because some of the documents um um are fairly short or too long for us to uh take in right for training and we actually use um a u classifier that's called a can LM uh to compare the documents with um Wikipedia uh articles and calculate the uh proxity score uh to understand whether the quality is good or not um a lot of URLs um are um copyrighted or having toxic content so we do a URL blocking as well and then the Hy analysis which stands for hate abuse and profanity uh so that's the main chunk of the analysis that we do for each of the document and uh remove or um uh delete the whole document if necessary the toxic uh content um and then as I mentioned earlier there uh personal sensitive information involved in the documents so we annotate and mask those as well um and then last but not not least um the the bias that's uh in the document that we actually anal analyze those and and um and doing some filtering accordingly as well and towards the end it's the uh um overall filtering and here the interesting thing that we have learned is um um in ideal situation you want to do all the annotations and you futter at the end but because so much computer is involved uh thousands of um uh uh CPU Calles uh are involved and in some cases actually we use a GPU uh for the U uh pre-processing um you actually want to optimize uh SE sequence-- wise uh before you do the final filtering so there's some uh interesting back and forth discussion and uh optimization that we're doing internally and then you feed the whole U data set into a tokenizer for tokenization and then that gets into the model so that's pretty uh interesting now on top of this in addition to making sure that the data quality is um good enough and uh risks are mitigated accordingly um there's a strong requirement that we haven't really come across before uh which is uh data Providence um in many cases uh people talk about uh data lineage uh that's another term the primary use case is given the uh public and government attention on Foundation models um we and others uh who are producing these models need to actually generate data and model cards um and sometimes they called Factor sheets um and it is to provide the full TR uh transparency and also traceability from uh data two model uh that's the forward lineage um up until uh some of the downstream applications and then being able to trace back as well right um and it's not just about the data lineage at the individual document level um that's something that we already track but also the job and the run that has been uh performed on the uh on the data itself so that become more and more interesting and important as well to disclose all the lineage because if you think about it any of the uh pre-processing jobs if they have um them uh code change and if we don't know about those code change then uh you will have some discrepancy in terms of the results that we produce now there are other use cases I think this morning there was a talk on uh open lineage uh that mention uh so you could identify actually popular data sets or even documents level um and have some return on investment discussion you can also start to have some debugging so if you find any issues Downstream with regard to the model output uh you want to be able to pinpoint um Upstream to the tokens but also to the raw uh data as well um and then last but not least the impact analysis say we procure certain data set from private Source proprietary source and that license has insired and we need to pull out that particular data set then what's the impact on Downstream models and applications so that becomes interesting uh exercise as well now on the data management itself um we actually went into a traditional route uh which is more or less a data Lake approach uh using a cloud object uh storage and simply because said um it's a uh relatively inexpensive to have all these data in one place um saved in a packet file format and it it can be accessed from anywhere uh with open protocol and then um in the case of we uh need to uh transfer the data from one place to another as long as they are in the cloud uh the cost is minimum but we run into a lot of issues um the very first one is uh whenever you want to make any modification on the pcket files in the cost bucket uh you have to save uh additional copy of that and because we are dealing with um um multi uh terabytes of data um that that becomes a a big issue even though the the storage itself is uh relatively inexpensive um another thing is is the Version Control uh that's uh lacking even though you can manually coordinate among team members to um control uh the access and also the version but it becomes really uh challenging at the uh uh fire level um and uh we have cases where um some accidental rights uh into um the uh Source uh bucket rather than targeted bucket um so a lot of accidents um uh for innocent reasons right uh take place uh that's not ideal um and then it's a relatively hard to track on the uh lineage and uh insights um because the data itself is stored in uh pocket files and you can only track down to the uh the file name and there's no way to get to another level at the document uh level and then if you want to understand what the data uh is and uh really explore the data itself um it's a little bit difficult to do as well so as it just happened uh last year also IBM is a um imp paral getting into the data Lakehouse um um space um and actually today there are quite a few talks on data Lake housee um and the whole idea is is uh uh to have the uh data warehouse uh uh performance but where the data Lake uh uh uh cost uh performance um level and then being able to have a open architecture uh so that our clients is able to leverage the uh the best or uh the better of both parts right the data warehouse and the data Lake um so we have taken that approach and and um when we look at the issues from the cause approach the um uh issues that I mentioned earlier uh we decided uh to also take on the uh Lakehouse approach um the core component here is uh on top of the cost buckets and the p uh five format uh we introdu um Iceberg which is a Open Table format uh now of course we have to to add our own catalog and uh lineage function but on top on top of it we are able to support uh uh the kind of uh execution engine of your choice usually what data scientists and data Engineers uh prefer to use such as um uh tributed notebook uh spark um Ray is a upcoming it's something that um uh R hat um uh as part of IBM is uh endorsing uh um so we need to support that as well and Flink and so forth and on the other hand um the lake house can provide insights right through um querer engin such as a presto or trino and then expose those insights and visualization in superet and any custom uh user interface from GetGo The Lakehouse um uh architecture actually allow us to avoid multiple copies of the data and because it's uh uh table format you are able to sort filter and a sample uh as you like so it has a friendly um connection with the SQL interface SQL queries that most of our data scientists and Engineers are familiar with and what's cool about also um the iceberg um uh component is um uh it generates automatic snapshots of the data um and we are able to actually provide tagging uh to um do the Version Control on the data itself and if we ever make a mistake we're able to do uh time travel um the tagging is very effective for us to uh manage the version end to end that I mentioned earlier in terms of the lineage and then the time travel really uh give a little bit more Tolerance on some of the uh kind of unavoidable mistakes that we may make right in managing the pipeline um now as we do this this is still very much a working progress what we found um also is a set of challenges and uh opportunities that we continue to um work on the first is I what I briefly mentioned Ray as a framework is a really a great framework uh that's scalable to run uh jobs in a Distributive fashion um but it's uh uh read and writer support is fairly lacking with Iceberg so we're trying to actually contribute to the uh Iceberg um uh open source Community uh with with the function that we have built ourself um and access control is uh not yet out of box with the Lakehouse architecture so we actually have to come up with our own um and the lineage is itself is so important for the foundation model pipeline um we actually invest quite a bit of time uh trying to come upate our own lineage and as I said it's not just typical data lineage that we have seen in the market um but also being able to capture the execution uh environment parameters and so forth that's really tricky um that's something that we are uh addressing as well and then uh the moment that we set up this um uh specifically for foundation model data pre-processing um there are other jobs if you like coming in as well and they come in in different shapes and forms and some of the jobs are uh pretty big and they deal with big data sets and some of them actually deal with a smaller data set such as evaluation uh finetuning and so forth um so the big question that we have now is uh we don't provide relational database out of our Lake housee but people are expecting especially for smaller data set uh the kind of performance that's uh available for uh relational um uh queries and and management so that's something we are dealing with as well um so this is at least to U us internally a rare opportunity to have everything coming together both the data and AI as well as the um governance and um and really think through how the data should be managed uh for a purpose right so that's something we're working on now um up until now I touch on the data component for the uh Enterprise grade uh implications obviously there are other things that uh uh we are doing to mitigate L risk uh from model point of view for example you can do uh quite a bit of what we call uh contrastive uh fine-tuning uh to have negative examples sitting along with positive examples uh to train the model um not to uh generate toxic uh content and there's a technique also called the rag uh retrieval augmented um generation that uh keep the model uh truthful to the um uh data that we provide to the to it uh so that it doesn't hallucinate on its own right giving uh answers to whatever question is posed to them and then use case is something very important both from actual impact but also evaluation point of view um but last but uh and the last but not least is the red teaming um and that's really in anticipation of um uh some of the uh offensive content or Pi content coming out of the model uh inevitably and then how do we address that at the model output level um and what's beneath it is really a layer of platform that we start to see and probably for the first time at least within IBM uh we see this as opportunity to bring data Ai and governance together and then we're introducing this to our clients as well as a way of managing their data and uh uh Ai and machine learning um and then of course the people behind it is important so the operations and the program management is also key um so with that hopefully I give you a um brief picture of what we are doing in IBM uh IBM research for foundation model training and um deployment and why we care about uh Enterprise great um um value proposition and then some of the that we're doing behind the scene for data management um so we have a few minutes for questions and I can stay a little bit uh if we need some discussion yeah we have about two minutes if anybody has questions that looks like a no all right can you talk at all about your data set sizes that you've we've got down there I'm I'm in IBM to so I'm Cur just curious yeah I well the data sets um I think what I mentioned earlier right the Comm CW snapshot that's a about a terabyte uh data but that's a already after the uh tax extraction before that it was much bigger yeah um and we talking about uh 30 to 40 snapshots and for some of the um uh data annotation it has to cut across all these documents so that's where the scalability and uh uh difficulty comes in right yeah let's go all right let's thank qu thank you