LLM Avalanche: Maria Vechtomova · Why is MLOps (and LLMOps) booming and how you can catch up?
thank you well very exciting to be here in San Francisco I'm heavily jet lagged I haven't slept for quite some hours from Amsterdam so it's uh yeah I hope my message will come over clear so I will be talking about why mlops and elements booming and how you can catch up and it will be about corporate setting how do that incorporate setting so I'm a lead machine learning engineer at ahudo Haze it's one of the largest retailers in the world with a lot of Brands 19 brands in Europe and in the United States and I'm leading mlops information there across old brands I have approximately 10 years experience in data analytics domain I've been doing all kind of jobs starting from data analytics to data science and machine learning engineering and seven years my focus was on mlops specifically so I've done data analytics data science and machine learning Engineering in large corporate organizations so I know a lot about politics not just about data science and machine learning engineering we also will be talking at data and AI Summits on Wednesday uh we will be talking about API deployment with model endpoints with serverless model endpoints from databricks and how we applied that we have not done that yet with llms but we are planning to do so in coming couple of months so as um was mentioned the introduction we do have a Blog it's called marvelous mlops so it comes because we are both me and my colleague Bashar both fans of Marvel and the mallops so we asked Chad gbt to come up with a name and it told us marvelous so that's how it got started we are really Marvel fans by wearing infinity stone ring and also bashack so are there many Marvel fans in the room yeah okay well nmlops obviously everyone right so that's cool so if you're interested to follow us and read about our journey follow please scan the code so what the talk is going to be about the story of a melops and Ella Lam Ops mlops adoption challenges and what you can do now to overcome them and llm mobs adoption possible use cases the challenges that are out there and how you can overcome some of them in the corporate setting so we're not talking about startups or anything it's really corporate setting which is very different um so what does Google Trends tell us it is actually funny I was looking at llm Trends and data science Trends and seems like LM was more popular than data science for quite some time well LM also stands for mastering laws apparently well it is funny people are more interested in law nowadays than they were before well of course it's about llms that got really popular after December 2022 everyone started talking about them including my parents which know nothing about AI so it is quite funny and you see that mlops is also really progressing a number of searches is really growing and you see some dips in in this graph and if you look what dips toes are it is actually Christmas period I don't understand why people don't care about the mail-ups around Christmas maybe production freeze um I don't know but it is funny but llmops is not a term yet so it still has to be maybe adopted more um so I guess llm operations are seen more like part of envelops yet still so what the envelope challenges do we have in large corporate organizations it's um hard to get buy-in for management to start with mlops if you if you just tell management I want to build the malops platform they will tell you well yeah what's the business value how how can you do that you need money so it's really complicated right scattered teams and not clear responsibilities across the teams and a change management is required if you look at the slide back they assigned started getting popular around 2016 so that's when it got more popular than llm and at that moment big corporate organizations started doing data science so that also happened to me I started working in 2014 as a data analyst and data science wasn't this thing back then at least not in Europe maybe in the US it was already a thing so data science job starts in appearing around 2015 2016. but then data scientists started bring building some models but there was no value coming out of it why because you have to integrate it with all the systems and data scientists didn't have the knowledge to do so and devops Engineers or software Engineers or other folks did have the knowledge didn't know anything about machine learning so that's why the boom of amalobs happened couple of years later so it is essential to do ml Ops to actually start seeing the value from from the data science um so but to do this mlops in the change management is required so what we see a lot is that well that's what we have seen at our when I started working there a couple of years ago that there was a one big data science team within one of the brands they were doing some use cases like demand forecast cross-sellment on the website personalized offers and it was a separate team they were there was also a team of devopsy Engineers there was a platform team they were not really talking to each other and if they wanted to get something done the data scientist they would literally have to send over an email with a zip file and all that crazy stuff and they will send back error messages and they it will go over and over so there is a lot of pain in this um pain from both sides so what can you do if you're in such situation so find people that feel pain so if there is no pain there is no need for cure right but if people feel pain you can address the pain and make them your ally that's what you can start doing you can start with one project that already has buy-in from the management and make it deployed in a reusable way so that you can apply whatever you used for every next project then you need to educate data scientists devops engineers platform teams on mlops and why it is important to follow best ML obstructices and speak loudly about every success you make so for example if you deployed one project in a reusable way every next project will be done in a shorter amount of time so example that I can give based on our experience within before it would take nine months to deploy a cross-all model on the website now it takes one month the model brings huge values like hundred thousand euro per week so eight months difference time times that amount per week it would be like 3 million euro and then you can tell everyone about it look we we just earned 3 million euro by doing this and the more projects you do the more the bigger the number becomes so it's easier to sell within your organization I think selling is really underestimated skill of machine learning Engineers data scientists to actually promote the work that they do and in the end you need to build a mailbox platform that scales and supports multiple use cases so mlus becomes part of enablement so then there are no there are no scattered teams there is a mail-ups team that fully supports all the all the projects and they are in constant communication with the platform team so in our situation for example we convinced the devops and platform team that we know what we're doing we got permissions to deploy ourselves and we build the whole framework that allows data scientists to deploy themselves within five minutes if they wanted to so it takes organizational change it takes talking to people all politics that people don't usually like but it's inevitable so um simplified mL of standscape can look in the following way so you basically need some basic steps here you need Version Control you need cicd you need orchestration model register container registry compute serving monitoring feature stores and there are a lot of tools here so you probably all have seen this mad landscape there are many many tools I don't think you need all those tools you just need to combine whatever you have in a smart way so I think that's how that works in the corporate setting for sure so and llmops now that does envelopes for lens require a new term well llms are more complex than traditional model so because it requires more interaction with the business teams you need more human involvement the constant feedback loop you need and people to annotate things it is also possible for more standard standard machine learning like NLP tasks but companies that were not doing that before on that scale they need to start that conversation with the business and you need to build something to do this feedback loop so it may require an extra additional thing in uml of setup to to make a lamp Ops possible and I think when we're talking about mlops we are talking about more platform thing the skills when we are talking about llm Ops we are just talking usually about deploying just one model for now because it's it's new so we're not talking about doing it on scale yet but it will come you see everything will come in couple of years probably based on the search terms in Google so what high value use cases are out there at the moment that we see at least from the corporate world so customer support I worked in a telecom company before customer support was always a big thing we are already doing NLP for um classification of chats and calls in 2016. and we already had this collaboration with the business there so they actually see the value of all of it and it's easy I think to to start with these kind of use cases for those companies for banks the same Insurance retail maybe some some of the retail it would also work shopping assistant for retailers it's an amazing opportunity for cross-selling crosstalk has huge value for retailers so this gives an extra opportunity and and actually makes kind of gamifications I think for for the customers so foreclosed Furniture Home Appliances and for food retailers recipes is a big thing so for example people want to make pasta bolognesia for dinner and they ask cha GPT for recipe for for this and it well much Egypt but this internal shopping assistant recipe assistant and will come up with products that you can add to the car directly so it's amazing opportunity for Grosso so one some of his examples that I did using chair GPT so for Ikea furniture so I asked GPT I have kids and cats and want to buy a sofa with removable cover it must be medium soft because I have some back pain can you recommend different sofas from Ikea rank them and motivate in Max 20 words so it comes up with some Ikea products I Googled some of them it looks looks okay so something like that would work for for a Furniture retailer right another example is recommending clothes so I'm a female going to a friend's wedding in Sicily and want to wear a petite size dress can you recommend some items to wear from zalanda.com it's like in Europe it's a big close retailer and accessories and shoes to go with it and it will actually come up with some suggestions well because charging is outdated those can are not sold anymore but I can imagine that for um for the Lando it's actually they're actually doing that by the way they announced in April I think that they're actually going to do something like that that's pretty amazing there is huge value in it but uh it's not always easy to reach we need to get buy-in from the business again it requires time and budget for fine tuning because you cannot just use uh open um AI for this or anything from azure you need human annotation and feedback so more close interaction with business teams and it takes time and it takes money available models don't work well for small languages so because our hotel Hazel works for 19 different brands we have brands in Europe and in countries like Greece Romania Serbia Netherlands Belgium so there are languages that not English so they they are very challenging and then you need to do something very custom to make it work and it it costs more money so what you can do already you need to get buy-in from the management so you can start with low hanging fruit examples like internal knowledge base that's something that we are going to do well everyone knows conference conference is horrible you cannot find anything on conference so scraping the whole conference scraping the whole GitHub for readme files and scraping sharepoints all together and taking Dolly and fine-tuning Dolly to answer some specific questions regarding your internal knowledge base that's something that we are going to work on because that's something that increases productivity doesn't have Direct business value but it's easy to implement it in English as well but it will show the business look we are capable of doing that this is this works amazing hopefully they're all very happy about it and it's it's a hype right so we can we can talk about it out loud everywhere um and and then hopefully we get we get buy-in to do more exciting things like recipes in Dutch so yeah that's a plan thank you [Applause] perfect so we have two minutes for one question someone wants to raise their hand if they have a question okay hey thanks for the presentation so um what model are you planning to use for your Internal Documentation llms and how are you actually going to train that model with your data for the recipes we don't know yet well I'm not really a data scientist I'm a machine learning engineer so I'm not really busy with training the models but we currently already have this knowledge base thing that some of our data scientists train so we are going to deploy that for recipes I think it's it's quite complicated because you also need to have we already have a recipe base so you need to fit that contact somehow so we need to build this system that maybe like some graph database you need to go in between and then feedback to extend the query something like that probably and also for the recipes it's hard because cha GPT can come up with some recipe with 200 grams salt in it it has no notion of taste so you need to give some ratings maybe to the recipes and maybe come up and it's also very biased because the ratings you have they're all very positive like at least three well they have no zero it should be really bad so you can artificially create these bad recipes with a lot of salt or something and maybe use that as well thank you have a couple more minutes what strategies do you employ to identify use cases like business use cases how do you connect with stakeholders to find those pain points well our scope is not data science machine learning engineering and usually we have business people that come up to us to the data science teams to deploy something so we we don't really we're not really busy searching for problems they come to us um yeah so we don't have the challenge really hi how do you overcome the culture that has probably the the data has not entered into the overall database and then how do we prevent that uh were the results constantly just perpetuating going to the English dominated culture for recipes for recommendations versus the cultures that did not mean recipes were not that they did not have the same access to the current database that's a good question I don't have answer for that I'm sorry [Music] all right last question okay that's going to be the last one there you go thank you great presentation I will stick to mlops here so one of the issues which we face with all these llms and deep learning models if you put in an app which is customer facing the response time is not you know the grade we need something like millisecond type of responses and I'm sure you also must be facing you know the similar challenges so are there any guidelines or techniques which you employ there well I think the way how people interact with the chat is very different than how they interact with regular apis so the response time requirements are very different because you can just it can start typing like GPC does so it is in that sense less challenging right but you need to agree with the business what is acceptable for them you also need to do some testing with actual users to see what is okay in your setting and it's hard to tell before because you haven't tried that yet so but I don't think it's a big issue just because it's a very different way of interacting all right give a big Applause for Maria thank you so much