SBTB 2023: Annie Talvasto, Kubernetes and MLOps for Scalable and Reproducible Generative AI
[Music] welcome to communties and mlops for scalable and reproducible generative AR workflows nowadays I think I got it quite okay practice makes perfect um but welcome what will this session be about so I'm hoping essentially because this is a bit of like a depends on where you're coming from what your background is on is this a beginner assist or intermediate or even maybe I don't know for some people it could be Advanced but uh My Hope Is that whatever your background is you learn more about how does amops uh AI generative Ai and all these topics relate to kubernetes how do these things work together so what is kind of the infrastructure behind the generative AI the large language metals and so forth um so whether your background would be more from the AI side or more from the kubernetes or Cloud native side I'm hoping hoping that we kind of learn together today what is the um you know match that these two make together perfect um okay so hi everyone I'm Annie tasto uh I'm CMO at Vision uh which is a Swiss devops company we have all products as well as Professional Services and a cncf sandbox project within our kind of uh you know under our umbrella essentially so I'm also a cncf Ambassador as well as an Azure MVP and what those two essentially mean that I am just super excited about all things Cloud native kubernetes and all these things and I just love sharing about them in events and such as this conferences to everyone who's willing to listen I guess um I'm also a kuber and CS cncf Meetup co-organizer since 2017 so few years now already I also coach early stage startups via different accelerator programs because that is my background essentially in in startup enablement and so forth and I'm also the co-host host of cloud gossip podcast um but I always forget mention something here so I've also been a a track chair for ccom plus Cloud native con I've been co-hosting the uh Cloud na live a cncf weekly show for since 2021 and so forth so um just all things Cloud native I guess makes me happy if one can say that good um so let's get started so we have an agenda for today um let's go so introduction we're doing really well we're already doing that then we have some definitions I think definitions aren't always the most fun part of it but I think in this topic in particular definitions are actually quite important to set the scene on what's Happening then we have why is AI or machine learning different from quote unquote regular engineering kubernetes and mlops qlow best practices and then some considerations because you do have some that you need to think about when you're doing development Within These topics and wrap up as well as resources in the end um but we don't want to we have to be having a bit of fun as well so you see here is a treasure map um as you can see it's a very like a regular pirate pirate treasure map you see there's the there's some dangers from the uh from the ship to the X which marks the treasure uh and what is the treasure it's scalable and reproducible generative AI workflows because that is the goal obviously that we want to reach today so that's why we want to reach treasure is these and we want to learn okay how do they happen how do we actually Empower these um and we have our little robot here that's going to be following the treasure map going along and we're going to be following that through all the hurdles of the mountains and the Sharks and so forth uh and what does scalable and reproducible generative AR workflows then look like or what is the kind of match there so uh everyone has heard about chbt open Ai and so forth these these are all um essentially products or so forth that are fueled by kubernetes and mlop so that's why we're discussing these things today okay let's get started introduction done we are already went our way as you can see our little robot is taking off uh so there we go we move to the first uh little jungle so what is that that was the definitions part so what is actually AI I think everyone kind of has an idea what AI is but what is it actually so it refers to the simulation of human intelligence in machines that are programmed to think and learn like humans um so it involves the development of computer systems and algorithms that can perform tasks that typically require human in intelligence such as understanding natural language recognize pattern solving complex problems making decisions and so forth then we move further within our jungle uh to machine learning so it is a subfield of artificial intelligence that focuses on the development of algorithms and statistical models that enable computer systems to improve their performance on specific task through learning from data and this is the key the difference here essentially it is learning and data is what you can kind of think about the keywords here compared to the previous one essentially as well as algorithms obviously and so forth uh so moving on to the next tree uh data science in itself is the multidisiplinary field that uses various Tech teques algorithms process and systems to extract insight and knowledge from data and actually have to say that it's quite fascinating that um with the like the boom of AI right now the hype that's going on actually job advertisements for example or searches for data science or data scientists have actually gone down obviously that has kind of Switched toward more like machine learning engineering and so forth but it's actually interesting that data science itself as a term might be not going nothing's going going to go fully out of favor of course uh but it might be kind of listening a bit and other terms that are a bit more maybe precise and descriptive to the ways that we're doing nowadays might be taking over a bit more um and then key part here is devop so it's short for development and operation so it's a set of practices and cult cultural philosophies aim to improve and streamline collaboration between software development teams and operations now why am I talking about develops because I think it's important to remember okay what is mlops then it is machine learning operations and it set of practices principles and tools that combine machine learning with practices of devops so if one were to say like super simplified you could say that mlops is devops with data um so we can never kind of forget that as the base there's even a lot of people spoke about uh having like a you know Mass LS hierarchy of needs type of structure where you have machine learning operations mlop structure of of what are the needs that what you need to master what you need to do and devops is the foundation of it all so that's why actually it's critical um like if you're looking for any of the beginner bestas practices devops is usually a good place to start so that's the kind of setting the scene there so we have learned AI machine learning data science mlops AI uh and devops and so forth so that's what we are talking about here today so now our little robot moves further within this treasure map to get to the treasure actually in the end and we're going to be entering the mountains here to learn why is AI and ml different so what sets it apart from regular engineering um so Ai and ML workloads and all of the development that goes into it is deeply data Centric in nature I've been talking about data a lot but you cannot avoid the topic when you're talking about the subject so uh the quality and the quantity of the data significantly impacts the per of the models so uh ML and AI Engineers need to focus on the data collection pre-processing and feature engineering which are not essential in traditional quote unquote engineering so you have to be data Centric and you have to consider a data if you put bad data in you get bad uh data out or or your results and and your models will not look good another thing how this is different is model complexity so in regular engineering systems are typically designed uh and built according to wellestablished principles uh and specifications and in contrast um ML and models are really complex um and their behavior is learned from data so Engineers you really must uh experiment with various models architectures hyper parameters and training data to really optimize the performance then iterative development is really big as well so because ML and model uh development is itself is highly iterative so Engineers can continually experiment fine-tune models and retrain to achieve desired outcomes so um traditional engineering might have a bit more linear processes and predictable outcomes uh and then scalability is another big one because when you're talking about the scale that data for example itself causes this it's absolutely massive there can be a big traditional engineering scale as well but from the get go usually the a and ml engineering requires you to focus on this one because it really requires scalable infrastructure to handle the large data sets and computational demands so it might involve um distributed computing framework GPU clusters and cloud-based resources now people sometimes ask me can you do this without cloud well I think everything and anything is possible if you want to set your mind into it then it is possible people have there examples of doing things without for example Cloud native U resources or cloud-based resources but the obviously is kind of a question of what is the easy and what is the hard path and if the hard part is required by you or your organizational needs then you have to go for it but um it might be a bit of a more Rocky Road um down the road monitoring and adaptation is super crucial as well so mln AI systems require ongoing monitoring to detect issues like model drift data quality changes and shifts in user Behavior so Engineers must build systems that can adapt to these changes in real time which is not necessarily need in regular engineering and testing and validation is super important so in regular engineering testing and validation can be based on um established standards and specifications it's a bit older and as I said it's a bit more linear so it's might be easier to approach there so in ml and AI engineering testing often involves evaluating model performance against a specific metrics so it might require domain specific expertise and may not have a clear-cut pass or fail criteria because it's a bit more complex in that way uh then another thing that people might not have to consider when they're doing regular engineering is explainability and bias so in the field of ML and II uh understanding why an AI model makes a particular decision is crucial and this is also obviously due to the ethical considerations um because if you are working in the field of um healthc care Finance or anyone that that matter that where your AI or your ml would be making any sort of decisions that involve people people you have to be aware what's happening and you have to make sure that you understand what is happening uh within your models um another thing that sets us apart is rapid advancement so the the field of mln is really evolving fast I think we have all seen the jumps that we have seen recently in in these fields uh with all the models coming out and and and people you know catching up on the developments and so forth so it is something that really affects the way the people are working on the field um and then inter interdisciplinary collaboration is a big one as well uh I'll talk a bit more about what does how does the production cycle look like but you have to really collaborate with different in uh different professions within the company or the organization that you're working for because it requires collaboration between data scientists software engineers and domain experts and infrastructure engineers and now this is I think a really a big one uh anytime I speak about these Topics in in conferences like I was speaking about in cubec con last week people usually come up to me and say oh I have this skill set how do I learn more about this other side or how do I interact and so forth so it's a big um kind of learning that we need to take on as a tech industry as well is to bridge the gaps between all the different professions that are working on these questions um and then deployment challenges is another one so deploying ML and a models into production requires special considerations so you must really ensure that the models are served efficiently and can handle real time requests and integrate with existing systems uh and cost is another big one anytime you watch any sort of like people talking about AI ml usually run into the topic of uh you know people are are talking about that okay we must we want to do this we want to do this at scale we want to actually inch the data but we don't want to go bankrupt for it and I don't think anyone wants to go bankrupt for it which is actually it is a consideration within this uh and if you look at for example the GPU prices and how they are going forth and uh I was just going through for example needing a GPU for demo and and pretty much the che even the cheapest resources are quite expensive at the moment if you need to get them so it's it's definitely something that you have to keep in mind with these in particular not that cloud cloud cost itself wouldn't be a topic in other engineering Fields but nevertheless so those are essentially some of the differences with AI and ml compared to regular engineering so if there is uh so these are some of the things that you have to keep in mind that are AI specific and obviously then when you think about the B best practices or how do we help with these these is some of the issues that you have to solve so you have to solve for scale you have to solve for the collaboration and so forth so these are also the talking points that people run into within their organization now how does it then look like from research to production so um it obviously requires expertise knowledge and time to build an AI model for sure but it might in some cases only take a few weeks for a a um engineer to come up with a new AI model or data scientist uh but then actually when you take it to production that might be months and months of hard work because they get getting to production is actually quite a big bottleneck nowadays so over I think the statistics are like over 70% of models never make it to production because that is such a hard phase to get over so it's the hurdle that actually kind of tramples most of the people trying to get the models into production and getting to see the benefits from it so it definitely is a bit of a work there and I think it's very telling that last week in cubec con for example the Keynotes within cucon were actually quite focused on like um maybe let's say Call to Arms for the cloud native scene to help with these bottlenecks because they are really issues that that the industry needs to be solving so that we can get to better place around these resources so it's definitely something uh that is important as I said to bridge the gap between these professions so why does it take a long time well it's because of all of the lot of the reasons why what I mentioned in the last slide so it really goes from you know getting something to work on your local machine as far as things go that might be relatively well I don't never want to say easy but it might be possible and then getting it from there to scaling it out to packaging up to to work on the pattern meters run scripts and so forth to figure out load balancing and data partitions and and so forth uh to fine tuning uh and everything so that everything is built for latency it all takes a lot of work and even even monitoring and logging for example uh takes a lot of effort than versus like regular kubernetes um because they in addition to container or kubernetes monitoring you have to monitor the data as well so it's always a bit of an extra consideration um and obviously you need to figure out Automation cicd workflows and everything as well so there's a lot of things that happen there but uh it's fine we can always obviously make it there so uh that is some of the kind of considerations why do these things differ and why is this such an important topic to discuss about infrastructure that builds all of these llms and machine learning models and so forth but why cuberes then I've been kind of mentioning it a bit here and there uh why is kuity such a good thing and then we're going to go as you can see nearby the water here uh we moving within our treasure map so cuberes and mlops why is it a match made in heaven or or so so uh kuun itself offers like a really kind of nice good uh lay base layer to build up on top of it and there's obviously some considerations if you're doing this with vanilla cuines I'm going to talk about Q flow a bit later on that helps you build upon on that pace but kuties works as a really good kind of um infrastructure to use to run your um containers in so portability is one of the reasons why communities works really well with AI and uh ml workloads so it kuties really provides a consistent and portable platform for running machine learning workloads so machine learning models um and their dependencies can be containerized using Docker making them highly portable across different kubernetes clusters and using even you can use even various CL different Cloud providers uh or on premises environment as I said before as well as well so you can really develop and test your ml models in one environment and seamlessly deploy them into another really without worrying about environmental specific issues as well customizability is a big one with cuberes so cuberes is highly customizable allowing you to tailor your infrastructure to your specific machine learning needs so you can configure resources networking storage and security settings to match the requirements of your machine learning workflow so customiz and enables Define grain control and resource allocation and uring that machine learning models receive the appropriate amount of CPU and GPU resources um performance is a big one as well so kuber is um is known for performance well it's for built for scale it's built for massive you know container orchestration projects and so forth and to be honest if you're asking me sometimes people ask me okay should I be using cuber un or when should I not be using cuber obviously always depends on your exact requirements and your setup but quite often if you have the scale that that might then require the kuun use if you don't have scale then you might not be in need of it or but as I said ML and AI itself are very um um scale focused to begin with because there's so much data so that's why kuber is really good match there as well so uh cuber is horizontal scaling for example so horizontal Part auto scaling allows ml workloads uh to automatically adapt to varying levels of demand and maintaining Effectiveness there as well and efficient Resource Management minimizes the resource contention ensuring that models receive the necessary compute resources to perform optimally as well consistency is a big one as well so kuber enforces um consistency by providing standardized and consistent environment for running machine learning workloads um and containers encapsulate the entire application stack ensuring that the uh same code and dependency be used for model development and um and user are used during the deployment then so it simplifies uh the reproducibility of ml experiments and results reducing the risk of dispares between development and production environment uh and microservice is a big one so um kuity supports microservices makes it possible that machine learning applications can fit into smaller and independently Deployable components which then really provides modularity and flexibility in m and workflows making it easier to update and scale individual components without affecting the entire application and you can build complex ml uh pipelines by composing and orchestrating microservices as building blocks as well and composability is the last one on this list of why this works really well nicely together so kuties allows you to compose and orchestrate ml components and services into end to end machine learning workflows uh so you really can kind Define share and reproduce ml workflows as code enhancing collaboration and Evy collaboration comes with again because these are all kind of interconnected topics always uh so by composing your ml workloads in kuber you can easily integrate various components like data pre-processing model training and model deployment into a cohesive pipeline so this is why kind of why I'm talking about kuber itself and as I mentioned kind of before jpt open AI most of the big models actually uh or all the services run kubernetes or different managed kubernetes services in the background so that's why that works nicely together then we're going to hop even more there are we going to go to the school which is the scary one I think or are we going to go somewhere else oh nope we're going to go to the next jungle uh we're going to talk about C flow which is the machine learning toolkit for cuberes so how many here has heard about C flow before two three four hands well that's good then then hopefully people are going to learn something new uh so qlow is an open source platform that that's designed to make it easier to deploy manage and scale machine learning workloads on communities and it provides a set of tools Frameworks and best practices to streamline the development and deployment of machine learning models in kuber's environment so cow essentially is designed to work on top of Upstream community so you can really easily use it there so it's a cncf project and cncf uh it's a cncf incubating project so cncf is a cloudd Computing Foundation that is uh kubernetes home for example um so it's it's fully open source it's integratable it goes into this customizable and extensible nature of cuberes as well uh and there's a lot of a host of other cncf projects that might want to use for for example for scale you might want to look into kada uh or monitoring the different PR me and so forth they're actually providing all the basic building box for your devop stack for communities but Q flow in itself is built for machine learning purposes so how does it then look like and what does it then provide so it provides that endend machine learning workflow flow that helps you actually manage these things on top of the kubernetes so from data pre-processing and model training the deployment and monitoring it helps you managing different tools and components separately so it provides scalability um and Resource Management as well so it leverages the kubernetes or orchestration capabilities and enables easy scaling U of machine learning workloads so that you can dynamically uh allocate resources based on demand optimizing Hardware utilization uh and improving performance it also helps with reproducibility and calibration another topic that keeps on coming up uh so uh Q flows infrastructure encourages best practices for reproducibility by capturing metadata environment details and code versions associated with each experiment so it really helps you manage from end to end everything so that you can really get the most out of your resources even though cubern is a good match Cube flow helps with that uh so this is how it looks like as well so um c c flow includes uh services to create and manage interactive Jupiter notebooks for example so you can customize your notebook deployment on your compute resource to suit your data science needs um you can use tal flow model training you can do model serving and you can use Cube flow pipelines to um deploy and manage into machine workflow so you can rapidly uh reliably do experimentation and you can schedule and compare runs and examine detailed reports on each run so there's a lot of that goes into it and there's actually really good resources as well from cow Excel where you can see before Cube flow this is how your uh stack would look like you know you have all of these faces from experimental pH to production phas that I was talking about before but with Q flow you can tap into all of these different resources so you see pie torch tensor flow as I mentioned before pipelines and all of these things um working together with ease within your um systems now an example of uh how this is what happening in production and in practice is shell they did a cube FL Cube K keynote a while ago I think it's 6 months to a year ago in in in EUR North America 2021 actually you can see it there so for them they have really seen a lot of benefits from using Q flow because they had a lot of challenges and these challenges might sound very familiar at this point in the conversation so they were challenges were disparate and in inconsistent Dev environments so creating proper de environment was tricky local data was they could use local data really well but you can imagine that with the scale of the data from with shell uh globally they needed a new solution uh they had a requirement for a lot of different specific specialized skill sets um and kuber nities requires a lot of knowledge and they were also wasting or or you underusing resources so they uh for them cost was a big consideration as well so CU flow helped them solve a lot of these issues with uh secure Cloud hosting and self-service so rapid onboarding at scale of was really important for them which is a big part of Cu flow as well um and Cube flow automated pipelines using kale as well and shared Cloud resources so they themselves said for example what took them maybe months before and now took them only days or or just a x amount of the resources so that was really helpful for them so you can obviously I for a reason I've attach the whole name here but I will have it in the resources as well but if you want to learn more you can obviously watch them say these things in their own words as well as an example of how do these things look like in production so perfect uh we're going to be then moving to best practices here to the mountains again uh and obviously I've been kind of running through a lot of best practices within this session as well already so this is going to be kind of like a compilation of all of some of them so it always starts with data that's what I said in the presentation as well already so if data is wrong you're not going to get anywhere you have to also kind of focus on mastering all the necessary skills to do these things and this goes into the the collaboration part of it so if you are coming from a cloud native background for example you might have to focus on learning more data Sciences skills maybe learn to P Bop on python even math on like statistical math and so forth might be something that you need to look into but if you're coming more from an AI or ml uh background then you might have to focus more on learning about Cloud native uh using uh clis and and bashel and commands and so forth umore your tools and when you need to use them so this goes uh abroad all across all of using cuberis as I said I just was speaking last week with a person who said that they looked into C flow and for their scale it did not make sense at that moment which is totally fine again all of these things are usually there's no need to complicate your environments if you are not at the spot where you need them but if you need them then you obviously tap into the tooling and use them when you need them for so from qflow to kada to different solutions that you can add on um to really kind of tap into it and obviously to all the uh cloud services uh Cloud companies Cloud comping companies such as like AWS Azure Google cloud and so forth and they in built um machine learning tooling so understanding the bottlenecks that you face is super crucial as well they're really focus on these to think about okay where do I have to fix and what do I have to focus on as I said mlops loves devops so you really can take a lot of the best practices there and using tooling like Auto mln and kaen is important as well so then we're going to go actually to the school we don't have a lot of time for the school I know that we are running out of time but we have some ethical considerations uh I trust that everyone will always focus on these on their own obviously so I'm not going to be diving too deep into here but you really do have to consider fairness and bias so you must address bias in the data itself and algorithms to ensure fairness you have to provide transparency uh so that we can you can trust and we can understand what's happening within your AI models you have to focus on privacy so you have to respect user privacy by protecting sensitive data you have to focus on accountability so developers should ensure that AI systems are accountable for the actions um consent and user empowerment is super important as well so you should have con user should have control over their interactions over AI systems security always important but particularly for sensitive data and uh machine learning and AI it is human Centric design as well so you have to prioritize human well-being and safety when designing AI systems data governance is important um as well so developers should establish a responsible data governance practices and long-term consequences so consider the consequences of these decisions that your models will be making as you go further as well as Regulatory Compliance that's obviously always important then um here is some links and slides so these are the learn more resources so I have them up uh I have the slides already up on my GitHub um and then uh I think the organizers are also gather at the slides so you can find them from there uh if you want to learn more like you know practical Hands-On examples how to get started on this on Azure Google uh or um AWS practical envelops by no and Alfredo is a really really great book that I highly recommend looking into that includes a lot of lot of practical while it's in the name tips um there's the shell keynote link there then there's some C flow resources as well as uh cubec con has a lot of resources around AI um day and and so forth so there's a lot to learn from there but that's it we're starting to be near at the end of the way we even managed to do the school uh uh School area with ethical considerations because it's very much make it or uh so far situation and then we're going to end up here in the Treasure which is great and here we are our little robot is happy we get the treasure uh off scalable and reproducible machine learning workflows for communities so that is really great we made it to the end uh we can keep ourselves a little kind of like tap on the back from making it to there and making it to the end but thank you from my side uh it has been a pleasure I hope everyone has a great conference [Applause] today