Bay.Area.AI—KUDO for MLOps: Kubernetes Universal Declarative Operator
Recording: Bay.Area.AI—KUDO for MLOps: Kubernetes Universal Declarative Operator
talk around 15 minutes on the design of TDC and what TDC is about and then Keshin will give uh Uh sorry guys, please keep stuff on mute unless you are the speaker. Um so, okay. Welcome everybody. Uh my name is Alexey Krabrov. I'm the founder and organizer of Bay Area AI and this is a joint meetup with the self spark reactive systems meetup kind of you know, I run a bunch of meetups in the Bay Area which are about general about distributed systems, software engineering and machine learning and machine learning from the angle of uh kind of scalable distributed systems angle not the kind of notebook, you know, it works like on some machine angle, right? So, we we we we're deeply interested in where the stuff runs. So, um uh D2iQ formerly Mesosphere is a long-time partner. They headquartered in San Francisco. The company actually replaces, you know, floor I used to work uh and run meetups and we did join meetups there
So, long history there founders. And um uh so, Kudo is the project framework which was I heard about it first from Ben Hindman who is one of the founders and kind of as a new direction. And so, from I think from our group's standpoint uh you know, a lot of uh machine learning is now understood on a single machine and the question is how do you run it on many machines. So, there's this whole area of ML Ops which kind of encompasses a whole bunch of stuff including how you run this, how you scale this, you know, how you deploy models and stuff like that. So, I will just, you know, give you a quick example. Like I recently got a book on Kubeflow by Holden Karau, Boris Lublinsky who I think is in the community now, right? In the chat and others. And so, you know, Chris Albon is the well-known data scientist. He has a uh is now director of uh machine learning at Wikipedia
And so, he in the uh in the forward, he basically says, you know, I know a bunch of data science, and so do you. And if there is one thing we need to learn more about is MLops, right? So, like this is you know, the set of skills which uh folks can add to uh the companies. And so, uh so we basically invited uh uh uh Victor and uh Jared Dillon, who's the creator of uh Kuda, to teach us about Kubernetes, operator framework, you know, all these new ways to scale things up. And uh I am assuming a lot of folks here are kind of uh there is a mix of data scientists, software engineers. So, maybe quick uh show of hands if you can use the raise hand feature in your chat. How many here are um uh call themselves uh kind of uh software engineers rather than data scientists. Let's kind of see where the divide lies. If you there is a little show of hand
So, okay, I see I see couple hands, all right. So, I would say yeah, Jared has two hands, so he's So, but but and kind of okay, so so I would assume, right? And let's say, how many folks here um deploy the service, right? Like like, you know, if you ever had to make sure a server runs something, and if the server goes down, your job is in trouble. Uh and you get a page, or, you know, you look bad. Like if you can raise your hand if you if your job depends on the server which should be up. And and you might be fired if it's down, right? Like that's kind of very simple criteria. So, yes, okay, we have the same same set of characters. uh basically. Got it
Uh okay, and and let's kind of I'll do the final poll. How many folks deal with Kubernetes in some capacity at their job? Like how many people deal with it? All right, so I would say about the same. Uh okay, yes, and and some actually some people love it. So, Boris loves it. Uh awesome. So, okay guys, so this is Jared this gives you the lay of the land, right? So, I think we kind of have a fairly newbie group of folks who want to take advantage uh of this stuff. And so, you know, please give us kind of an intro assuming people know basics of Kubernetes and they're not advanced, right? Advanced uh users. So, and with that introduce Jared
Jared, please tell us about yourself uh and and uh you know, uh tell us all about uh Kubernetes and Kudo. I'm looking forward to it. Great. Hey everyone, thanks for taking the time out to uh to come uh listen to this. Um this is this is fairly informal. Uh you know, there's uh I I've given I've given forms of this. I I think this is kind of a nice um blend into reality for for uh Kudo and how it how can affect the real world. Um for whatever reason my Zoom chat is is not opening up
Um so, if anyone has any questions, Alexey, will you just interrupt me if someone types a question. Yeah, so uh folks, feel free to put questions in the chat and then you Jared can see the chat or I can uh you know, bring it to your attention and obviously at any point feel free to just declare a Q&A option and people can just ask questions. Great. Okay, actually, you know what? It was behind my screen share, so I found it now. I'm testing out Windows Subsystem for Linux, so uh so nothing I'm normally on a Mac, so everything's really unfamiliar for me at this at this moment, but uh uh it's kind of the only way to get like 128 gigs of RAM if you're working on Kubernetes locally, so um Okay. So, yeah. Hey, hey everyone. I'm Jared Dillon
Um I'm the creator of Kudo. Uh Kudo is short for the Kubernetes Universal Declarative Operator. Uh it's a low-code way or or operator-friendly way of of building out operators. Um here's a whole bunch of stuff. We're on the Kubernetes Slack. Um there's some demos. I think that demo's actually old. I think what we'll see today is a little more up-to-date
Um We have a website. We have a Git repo. We have like something like 850 stars. Uh healthy CNCF sandbox project. And we'd love to talk to you about running uh machine learning workloads on top of Kubernetes today. So, quick agenda. Uh I'll introduce myself real quick. Uh just just as uh shortly and and without any uh you know, any any any build-up
Uh we'll talk a little bit about Kubernetes and machine learning. Uh I'm a I I'm a distributed systems engineer. I'm not an ML engineer. So, I focus very heavily on what people want to run on Kubernetes. Uh I am not an I I I uh I you know, do not work in any sort of training serve you know, serving, data preparation, any of those other sorts of roles. So, from a specifics of Kubeflow or anything else, uh I would defer to those experts. I'm only focused on how to easily run tools like Kubeflow on top of Kubernetes. Um You know, I want to talk a little bit about stateful workloads in in Kubernetes
Uh 2021 is is a continued increase in the number of stateful workloads we see on Kubernetes, but we're still in the early days. Um and we'll talk about some issues with that related to Kudo. And I'll do a quick demo, and then we'll go informal chat after this. I can show off CLI stuff in the uh uh in a screen share. We can just have discussions, whatever else. I don't want to uh spend too much time in slides. Um probably hopefully hopefully about 30 minutes or so in slides. So, yeah, who who am I? Um I'm I'm a systems engineer
Uh I happen to also like distributed systems. Uh those those two things are separate, but talk to me about, you know, Rust on one day, talk to me Erlang and Elixir on the other day. I I I you know, I but as far as orchestrating workloads, working on systems, that is uh the things that I work on. Um and and how that blends into structure. So, my So, my call it an infrastructure engineer today, uh but I but I'm very much a software engineer first and working on these sorts of problems. Um I've worked on Kubernetes since uh pre-1.0, since about Epic. Um early developer, contributor, and user of it. Uh and I'm very active in SIG API Machinery and uh CNCF SIG Application Delivery and the Operator Working Group
Uh there's a leadership changes happening there, but I'm you'll generally find me in various Kubernetes and CNCF areas uh continuing to contribute uh especially towards workloads on top of Kubernetes. Um and how we advance the state of that and and programming models around Kubernetes. Uh early contributor to Helm uh 1.0. Uh I was I was uh I co-helped co-create that and I created KUDO, the Kubernetes Universal Declarative Operator. So, uh I've been I've been working on Kubernetes now for uh 6 years. It's 2021-ish, six-ish years, a little little under, and um and I focus on this very heavily. So, happy to answer any questions. Um so I I you know, I I added a couple slides mostly to frame this talk around why uh uh you know, Kubernetes and machine learning, Kubernetes stateful applications is important
Um uh and then we'll kind of like get through this and get to the meat and to what KUDO is, right? So, uh there is a uh a growing need or growing demand within companies to effectively run everything on Kubernetes, right? And that that includes machine learning stacks. Uh there there's a lot of services out there. There's a lot of ways to to run the tooling that that's all goes into building and serving models. Uh but there's an operational demand out there in the marketplace for actually unifying everyone's stacks. And so I would call this in a very early space, right? We're we're about half as long as in this as we are uh Kubernetes and shorter if you think about containers as a whole. And uh but it but we are starting to mature. We're starting to think about the full life cycle of applications, right? If you If you think of machine learning, you have a whole bunch of steps and everywhere from data ingest to munging that data to training your models to uh to actually serving them in your applications. And these are all like different phases of maturity in that I would say
Uh Uh so but but we we do have some really nice tools that are building upon that, right? So So we we we have this wide focus around a uh a particular use case, right? So if we look at like Kubeflow out there, it's really leading the pathway to ML on Kubernetes. Um And and D2iQ has something around that called Captain. Uh you feel free to look it up, but it but it's it's an operator written in Kudo on top of Kubeflow. So we see the demand out here. People want to be able to have this as a one-stop shop uh to run all these tools uh it it on top of Kubernetes. And and there's there's something really important about this that we'll talk about in later slides is it's not just the power of running Kubeflow in your production infrastructure. It's that you can lift and shift this and run this on your laptop, too, right? You can run this on your desktop computer. Um and you can do it in the exact same way that your ops teams are doing it in production
Right? So so one thing Kubernetes and Kubeflows is has enabled us to do is remove that impedance mismatch in your tooling between all those environments. And I think that's a really powerful property. Uh we we we often very much focus on the software itself, maybe or the releases around the software itself, but we do a lot less focus around the ecosystem around, right? And uh you know, raise your hands if you've ever been in an environment where you know, in production it it had this beautiful CI/CD pipeline and your development environment was just a mess, right? You You may If you were in an advanced org, you you'd you'd clone a repository and run make and it might work for half your new employees, right? Um one one big benefit that I think goes goes unrecognized is uh with with with Minikube or kind, you can run the exact same same environment in your on your own environment and know exactly how it's going to work. So okay, great. Uh okay, you've you've you've convinced me in your 20-second discussion about Kubernetes and machine learning. So I have Kubernetes now. Why, you know, who cares? Like what do I do now? I have Kubernetes now what, right? So uh over the past 6 years of Kubernetes, over the past uh time since Docker came about and all that stuff came about, um stateful applications have sort of been this unanswered question or this uh uncertain background around moving all of your workloads to Kubernetes. You might have your web servers running as containers and then run your database, right? Run Kafka, run everything else at on on bare VMs or use managed services or whatever else
Um And even in 2020, uh the CNCF survey report for 2020, we see that while we're falling in the minority, about half of people are still running stateful workloads outside of Kubernetes, right? Uh and or or if you want to flip it, only about 55% people are running stateful workloads in containers. Um if we were to to go with the the postulate like if we postulate that the vast majority of production containers workloads are on Kubernetes right now, um we can we can we can extrapolate that well across the entire containerization industry, right? So, about uh a little over half are using stateful workloads in production. And there's some reasons for this, right? If we think about our stateless workloads, we think about the actual apps serving these, the API serving these, they're fairly uncomplicated, right? They're that's a bundle of code. Um it's probably calling out to some stateful service that we'll get to in a moment, but by comparison, it's it's it's it's serving a request and response. It might be listening on a queue. It might be performing some actions, but largely our front ends are fairly stateless, right? Uh we we we don't worry too much outside of maybe sessions where any state in the actual application process is. And uh And so, when you start to tackle So, so it's it's it's actually to to to close it up, really easy to throw a stateful uh API onto Kubernetes, call it a day, right? Stateful application life cycles get a little more complicated. Uh you know, if if you look at it at the tooling out there for Kubernetes today, we'll talk about this a little bit more, but Helm, running cube CTL apply, uh a lot of these really only focus on the part of the life cycle where you deploy
And if you need something that is persistent, right? The this gets into the the pets versus cattle uh description of distributed systems. And for those who aren't familiar with it, uh you know, if you have a pet, you're going to name it, you're probably going to take care of it when it's sick. We typically use this to refer to databases. Uh and and versus cattle, which we typically refer to right now to to refer to uh stateless applications. Something's going wrong with your Rails app, you kill the process, right? You you you put the cow to pasture, and uh new cow comes and takes over the work of that old cow. So, we built a large amount of skill set on dealing with these stateless applications. Now, as more and more organizations are moving to running stateful applications, tools like Helm, tools like uh just kubectl apply, uh we you reckon like we start to recognize only really fit on the deploy part of that life cycle, right? Whereas stateful application like life cycles are very complicated. Our application's first direct interaction with our database is, other than storing information in the database, is migrations, right? How do we How do we preserve a data model that's up-to-date at all times? Okay, great
Sure, fine. We'll we'll write something for that. Uh okay, well, what about upgrades, right? And and upgrades is a very tricky thing, right? We you you have some upgrades that are very straightforward. I go from 2.2 to 2.3 of Kafka, everyone's happy. Things generally work as we expect them to. Until we get into to a uh for example, an on-disk change, right? You you move You move from major versions of tools and databases that actually some sort of human migration step in order for it to work, right? Uh and so now it becomes a little more complicated. So, upgrades become a very complicated topic, especially since you can't just take the same database file in a lot of instances and say upgrade from Kafka 0.9 to 2.7, today's today's current version. Um and so that requires an operator to do
That requires a human to do, to go understand and figure out how do I manage this upgrade without dropping any any messages that are coming into my topics? Or how do I how do I upgrade Cassandra without dropping any data? And accept as many writes as I can along the way. Um and then how do I handle backups and restores, right? As as people uh tend to discover, just snapshotting an EBS volume or cloud volume is not enough. Um there's plenty of reasons why that fails, but it comes down to the the application itself is the only thing that can understand the exact state of your data, right? And there's many reasons for that. Write-ahead logs may not written to disk. Uh fsync lies with Postgres and EBS especially. There's a there's a massive issue about that in the in the past, but even backup and restore isn't as simple as it is. And so, what we're what we're starting to settle into when we start talking about stateful application life cycles is uh not not something that is is fully generic, right? You can't just apply the same strategy to every single stateful application with these things. You're really talking about some sort of application awareness
Right? And and so, how do we how do we, you know, on one side we have our our cattle. We actually don't really care if it's a Rails app, if it's a Node app, if it's a Go app, if it's Spring Boot or whatever else, right? It's a piece of code that's running, it's listening on a port, and it's black box. Uh and we generally only have one pathway to that box, right? A network port. Uh we we open it up far more when we have stateful applications. And so, uh how you handle that in containers is is a tough deal, right? You're If it If a container dies, you usually just replace it. Well, what happens to the data? What happens if I have in-flight migrations, everything else, right? Um so, in in like I said, standard tooling orients around the deploy part of that life cycle. Great. I I I spun it up, we're done
Um you know, we we don't worry about the complicated operations that happen after that point. And that's just not true, right? That's where That's where a lot of tooling fails to sort of stand up to that. So, how do we get to better stateful applications? Well, uh for those of uh for those who are uh less familiar with Kubernetes, there's a very important tenant that we can take on uh about Kubernetes or property about Kubernetes, um that that sort of reveals its power. Um and that is and that it is reconciliation-driven, or as we might call it in in signal processing, level triggered rather than than um event triggered. So, the difference is is API isn't Kubernetes and and and software Kubernetes in the Kubernetes ecosystem advances based on a a desired end state. Not the fact that I want to make the change, right? And this is a very subtle difference. Um instead of saying scale to three, I tell the system I desire three replicas, and the system figures out how to scale it for me, right? So, if I'm end user of some sort of stateful application doing training, doing machine learning training, and I might be interacting with Kafka as as my one of my first data sources, right? Going into Cassandra or any else. Um in a traditional environment, I might have a Kafka instance, I might have to think about writing backups and restores, I might have to have a scaling runbook and everything else
In a reconciliation driven or level triggered world, as the end user, I just ask myself, well, how do I make that as declarative as I can, right? Starting with the version of of Kafka that I'm running in this example, right? And then I stop asking I stop asking to to take backups. I I I'm not in like but I start I I I start telling the system, "Hey, I want backups that are no more than 24 hours old or 6 hours old or 1 hour old, right?" Um and then boom, give me an imperative restore status. And then even number of brokers, right? If you if you go to scale uh a lot of these tools, uh especially Paxos raft driven ones, scaling's not just a matter of increasing your number of replicas and calling it a day. In a lot of these systems, you actually have to uh accept the new member as part of the cluster, right? Let let let data let let all the shards of data sort of rebalance. Uh let the let these cluster distributed systems operate as planned with all the hooks and the extra steps that are needed in order to for them to guarantee consistency, right? We we want You know, one one one interesting thing about containers and stateful driven applications is uh scaling the number of containers or the number of replicas doesn't necessarily mean up like doesn't necessarily mean in applications like Kafka's or Cassandra's CAP guarantees, right? You you actually have to run things in that application. Okay, great. So, what if we introduce another layer here? What if we introduce something then to take advantage of this dec- declarative definition and actually start to execute actions for us, so we don't have to execute those actions for ourselves. And this is where operators come in
Operators were originally written about in a CoreOS blog post, um as they were launching their etcd operator, but really what we're talking about is taking those that runbook of operator skills to achieve a desired state and building that into software, so that this is so that the steps around achieving the uh those sorts of actions that we want to perform or those states that we desire, we want something to actually perform those actions repeatedly over and over and over again and reduce the the uh uh human element in in in sort of these uh standard workflows, right? So, if we look at that that that leads to maybe a Kafka operator, right? And we translate that declarative definition into imperative action. So, we're going to go upgrade Kafka images. We're going to generate out TLS certificates. We're going to back up and restore your data, right? We're going to uh automate your DR process. And we're going to do that based on the definitions and facts that you set out uh from your your declarative definitions. So, operators have been in in play now for a couple years. Uh they've been here, but there's a lot of challenges, right? Um imagine being the person who not only wants to use Kafka, but now needs to write some sort of operations framework for it, right? You need very advanced Kubernetes and and you you need a whole bunch of knowledge. You you need advanced Kubernetes knowledge, writing controllers, writing custom resources, you need distributed systems knowledge, you need knowledge about the thing you're building, right? You need If I'm writing Kafka operator, I need to know everything about Kafka
Um and the profile we see is we typically start talk to people who are who know a lot about Kafka, but don't want to talk about Kubernetes as much, right? Or or they treat that as a deployment target, um not its own thing. Uh so so that's So so it's a building block on that a little bit, right? Um Anyone familiar with the Kubernetes space, most of it's written in Go, right? And uh and so that that's that's challenge one. There's there's some frameworks in Java, there's a framework in JavaScript, there's a couple others, one in Elixir, but primarily this development happens in Go, right? Um and it can be very challenging for for people in the space to actually go and hire for these skills, right? If I'm If I have my own internal uh ML pipeline, I don't want to go rewrite all the hand processes that I do today or my bash scripts in Go and learn how to do that on top of Kubernetes. And building a really good operator in in in Kubernetes takes tens of thousands lines of code and months if not years. You end up code duplication, you have a high maintenance burden, um you know, the the there's there's typical pathways here that sound great until you actually have to go do them and and you're trying to live uh like uh when when you're trying to build the software, it can be very difficult to get everything you need to make those sorts of projects successful, right? Um and what we come to find out is running stable workloads in Kubernetes way too complicated for many users. And even if you're just going to go out and use operators today, they have completely different workflows, different APIs. If you're going to go run Zookeeper, Kafka, Cassandra, Jupiter Notebook operator, everything else, you're trying to figure out how to operate those all in different ways. They're inconsistently debugging, and you have a whole sprawl of software that runs your software on top of Kubernetes, right? If you If you consider the the fact that if I have a staple application and I've bought into this pattern that I need an operator for it
Well, that scales and you know, end to end for however many operators you have. And so, all those may have inconsistent services. Now you have 15 operators to deal with on top of 15 different data data stores. And that's something that we want to avoid. So, with that, I want to introduce Kudo, the Kubernetes Universal Declarative Operator. It's a toolkit and runtime for building operators that are really optimized for these complex staple applications. We're talking Cassandra, we're talking Jupiter Notebooks, we're talking uh Spark, right? We're we're talking Airflow. Uh all of these multi-component systems that have a couple properties
One, they're distributed and and and complex in the um simplex versus complex sense, where uh they they all they all interact with each other, right? There's no one consistent braid. And they compose. Right? If you talk about a full ML uh serving platform, right? Training and serving platform, you have a whole bunch of components there that all need to interact, right? So, Kudo is trying to optimize towards that sort of software. And we want to increase developer productivity when building those sorts of operators. We um To put it another way, uh if you're an expert right now in Kafka, we want you to be an expert in Kafka, not an expert in Go, Kubernetes, and Kafka. Um And we want to and and it's very focused on increasing operator productivity with operating these end services, right? Give Give everyone using Kudo-based operators a consistent uh control plane to which they can manage all these services together. So, why Kudo? Uh building operators is a very hard work. It requires a very deep understanding of the Kubernetes API
Um It requires a very deep understanding of all the work and everything that we we we we put into building operators over the past few years. And all the languages and and and taking and and learn how to take advantage of the momentum that's built up in all the tooling built around Kubernetes over the past couple of years. Um I don't think people uh appreciate this enough that, you know, as as against I I'm I'm fairly against code generation, but it's hard to argue with how strong the ecosystem is when you can just go pick up a whole bunch of types and run with it. Uh the problem is if you don't know go and you don't really want to write go all day, um you know, I I I I I I I talk about this in in another format, too, where uh if everyone built their own operators at this point, we'd all be experts in Kubernetes, but not the thing we originally set out to build the software around, right? So, we all don't want to become experts in Kubernetes. Uh some, you know, people have different interests, and so uh we need to find an easier pathway to actually build these operators. Uh maintaining these operators is really difficult. So, let's say you've gone through all this work and you say, you know, I just really I'm going to build build an operator in go. I'm going to use controller runtime and everything
That's all great, and then a new version of Kubernetes comes out, right? You have to deal with all the deprecations, you have to deal with all the new code gen. Uh typically takes when we update Kudo about a week when we when a new Kubernetes version comes out to sort of suss through all of the changes that happened to enable them for our users, right? And so imagine losing that time for every single operator. Uh And and I think this contributes very heavily towards the uh the lack of the sort of software out there uh because the the building and maintenance is still hard. And then operating these services is very hard, right? Like um there's no really existing solution that is turnkey that lets people easily operate and deploy services out in production and have a similar control plane for everything they're running. Even though the mechanisms are different, we still want to perform the same sorts of actions on our data stores, on our ML pipelines, no matter what they are, right? And every organization's going to be a little bit different on that. So there's a ton of value in making this this a lot easier. So uh you know, here's how KUDO helps developers for you know, build people building operators if you're going to go build um your ML pipeline um components on top of KUDO. Um we provide abstractions for sequencing this complex software uh so that you know, you can conceptually take them as similar to to run books
Um it's it's a sequence of steps of of how do I actually deploy this application, right? Uh It it it the there's there's some great examples as you go in uh our operator library. I can show some off as well. But uh for example, you can separate out uh the fact that a certificate needs to be generated before you even stand up the the software itself, right? And uh and and pass information between steps. So there's a little bit of that workflow engine uh that enables uh all these steps that you'd normally find in a getting started guide. Um we're reducing boilerplate and code duplication between all these operators, right? If your entire ML pipeline and you have six different operators, that doesn't help you. But if you can maintain the operations for that under a single pane of glass, um that's a that's a pretty big win and saves hours per week, right? Hours every time you need to touch one of these components. Um reducing the number of controls in a cluster. I think this is only important if you're a Kubernetes administrator and trying to like trying to operate very leanly, but you reduce the number of software maintaining your software
If you if you think about an operator, you need one one operator for every type of thing that you run. Reducing that is a good thing, right? Cuz you don't have a Kafka operator, you don't have a Cassandra operator, an Elastic operator, a an Airflow operator, everything else. You have one control plane sort of managing it all. Um We have extension mechanisms. Uh probably won't talk about that today. Um we want to be able you know, I think I think this this is a huge value proposition. Uh When you pick up a lot of these tools, you go to meetups, you get a bunch of uh you you you look at a whole bunch of different manuals, best practice guides, you try to figure out how to actually run it. Uh and none of that really gets you ready for your first outage with that, right? Um Or or gets you into the like how do you know you're production ready? Uh I I think what I like about that a lot is the you know, the way you know it's production ready is either a a company's going to tell you tell that to you or you have and and they're going to own part of that, right? You're paying them
Or you're going to sort of get a gauge for that on on tribal knowledge. Um if you can just incorporate this in an easy-to-go tool uh and and you make that accessible to companies, then you can ship your best practices for these operations alongside the software, right? Like building that into the operator is a net benefit because it's it's something that is and now you you've taken an entire runbook, you've taken documentation, you've taken uh solutions architects and and you've you've taken a common pattern and made that repeatable, right? You you you can ship runbooks alongside their software and I think that is incredibly powerful once you can start to to to do that and really focus on what does the end user want to do, right? Um and we ship with a testing tool. I won't talk about this today, but uh we we ship with a testing tool for TDD of Kubernetes resources. It's called KUDO, kuttl.dev. Um I think Boris uh you know, especially like KubeWay might be interested in that. We're trying It's also CNCF and um it's a really great Kubernetes black box test tool. So, how does Kubernetes help you KUDO help users? Um yeah, we we pre- we provide a plugin for Kuber- for kubectl to deploy, debug, manage workloads. Um and this is your one-stop shop for managing all of your operators
They're all based on custom resource definitions. Um and we're working on ways to centralize more of this. Um and we we we have a lot around metrics and alerting story and security and our back, but really starting to centralize that so that you have a single pane of glass for all of your operators is where we're going for for manage continued management of all these things that you're running on top of Kudo. Um let's see. Uh I think I think all I'll mention here is that uh so the way we break out uh life cycles in Kudo is you have a series of plans. So you might have deploy, you might have update, you might have uh you know, rotate search. So we have a couple that are built in. Deploy, update, and upgrade are all built in
Um and represent base CLI actions, but arbitrary plans are supported so that if you need to do something like backup, you can write a backup plan, right? And that's the intent there. Um these can happen in phases, and then phases break down into steps. And where why this becomes important is uh is let's uh this this this gives you two forms of of uh workflow management, one very granular and one uh much more coarse. So from a coarse perspective, coarse in that coarse management, you might want to go generate out a a TLS cert as a phase and then deploy with that TLS cert, right? Um in your steps, you might actually deploy out your entire application. So uh you two different grouping mechanisms, uh you could technically move this into one grouping mechanism. Every time we find out from a semantic level, having this phase, this coarse-grained phase, uh ended up being a a a uh useful decision. So if you're building operators, you have a you know, couple levels of granularity there. Um Let's skip that
I'm going to go to the architecture. Uh so if you're using Kudo, your your primary interaction is from the desktop. We have a cube CTO Kudo plugin that I'll show off in just a moment. And what it does is it interacts with our operator repositories. These look a lot like helm chart repositories. They're easy to air gap. They're easy to put into your own environments. Um but it's ultimately a library of these operators
And then from there that that we get that operator and we deploy that into your cluster and have a whole set of CRDs from operator operator version to instance that ultimately is an abstract representation of what you want to deploy. And I apologize. I have to bio break real quick and I'll come right back. Can we take a 5-minute break, Alexi? For sure. Absolutely. Okay, I'll be right back. Boris, I wonder if you're here. Maybe we guys can have some kind of chat meanwhile if you're so inclined
Um so I'm personally rather new to Kubernetes. I wonder if anybody can um share So I'm curious, right? Um when a new version of Kubernetes comes out um something like Kudo is updated and uh I'll ask Jared as well when he comes back, right? Like what what are the things which are pain points for general developers which things like Kudo abstracts from, right? So, what are the kind of fast-moving pieces of Kubernetes which change now which are pain points for developers if folks have experience with it? It's not uh actually Kubernetes. Uh Kudo doesn't really help there. It's uh changes to the application where operators help. Ah. Uh it actually a little bit worse because with the new releases of Kubernetes, the APIs will slightly change. So, all of a sudden, operators, some of the operators are broken. Ah
All right, I'm back. Uh Jared, yes. Meanwhile, I asked a question. Maybe you can uh share your perspective. So, I was curious, you mentioned that Kuda uh kind of protects developers from um new releases of Kubernetes, right? So, it takes you much faster to adjust it to new releases than, you know, each individual developers would. So, I was asking, what are the first moving pieces when a Kubernetes release happens? What do you Like, what What does it entail, you know, that like week for the tuning Kuda to the next release? Yeah. So, you end up with a with a diamond dependency problem, and I can I can just draw it on these slides real quick cuz let's imagine um you know, we we have you know, I'll just open up a new slide. Here we go
Um so so we have a Kubernetes version. I'm just going to write I'm not I'm not going to make it look pretty at all. So, we have a Kubernetes version, right? And um this is These are like the core libraries that get tagged in the in the Kubernetes repo itself. But uh what happens is is you have a whole bunch of software that depends on that base Kubernetes version. So, you have a client, you have um uh like the API machinery sub package, a whole bunch of stuff that's moving out into other GitHub repositories. The problem is if I were to do some arrows here, and I'll even add a controller runtime, because I work on controller runtime as well. The problem is you end up in these diamond dependency issues. So, let's say Kubernetes is updated
Um these are all dependent on core Kubernetes libraries for data types and other things. So, if I have my operator down at the bottom of this whole um chain, the problem is is my operator might be a dependent of any number of these things. And so, I can form diamond dependencies, right? Let's say Let's say client goes updated to Kubernetes 1.2.1 or 1.21, but controller runtime is still behind, right? Now I have a diamond dependency issue um that I have to now go sort out for every one of my operators. If I'm using go modules, I might have to use some funky replace blocks. Um it can be very messy trying to make this dependency tree work. Uh and so, one nice thing about about using Kudo is that you're not even you're not even thinking about this dependency tree, right? Your Your current version of of Kudo's going to work. When you update Kudo, it's going to work. You don't have to go upgrade your manifests because of a Kubernetes update
Now, that's not entirely true. Like, if there's a a a resource type that has changed its version, and you're using it as as an operator, you you'll need to go change that. But you don't need to go change like API level code, right? You don't You're not going to go muck about trying to solve this diamond dependency issue. Uh Boris, does that kind of uh does that land I didn't I it wasn't here for the answer your answer, but did that Uh my answer was uh slightly different. My answer was that uh operator implementation absorbs uh the changes in the Kubernetes. So, it minimizes the changes for your software. Yes. Yeah, no I I I think we're on the we're we're on the same page, so yeah
Um Ku- Ku- Kudo sort of like Kudo takes all of this, right? And then creates an alternative implementation uh it it creates a data-oriented implementation language for what you might write in Go, right? So, you're you you In this case, this would look like Kudo here, right? Dealing with all this, and then you have uh you know, Alexy's cool operator depending on Kudo. And uh and and since you're not using any Go, we can we can we can control this this issue at the Kudo level. Uh sometimes it causes a mess, but it's uh it's far better than having to do with that that five One specific uh example I have for this, by the way, um and I don't know if anyone else here has uh has has managed dealt with this, but uh uh when the the Kubernetes Go clients added go's context in as a as a um a parameter. People wanted up to it so that needed to change just about every single one of their call sites in function signatures to handle the fact that now every single API call took a a a go context. And so, imagine doing that if you're running five 10 operators, imagine doing that over and over and over again. That same sort of upgrade on all of your software. Um those sorts of changes can be very painful. All right
Um any other questions or or I'll continue and Great. Okay. So, uh we have a a CLI. Uh if if you look on the Kudo site, there's installation instructions for every platform. Uh but it but it's a uh a a cube CTL plugin. Uh you can initialize it, you can install packages. You can look at the plans and how um how the graph will will sort of get resolved and and uh steps will occur. You can see it see how that ended up happening
You can see the status to make sure that everything's up and running and and working as expected. Um we have an open governance model. We use caps. Uh we we really are about all ease of contribution. We want people to contribute. We want people to be part of this thinking about it. Um we have a bunch of reference operators. We have we have tons of releases
This is out of date. We're we're you know, we're a decently popular CNCF sandbox project. Um and and and what we're doing as part of the sandbox is is trying to better understand in how people run staple applications on Kubernetes and adapt to that, right? So, uh we we want to be the best framework out there for rapid operator development that keeps people focused on their own uh uh domain expertise and not having to learn Kubernetes and go as a sole separate thing as well. Um I'm going to skip some of these these verses. I'll I'll post the slides so that everyone has them. I just want to be cognizant of time here. Um we have a bunch of operators. Um we I think I think this is a repeat, right? Right? We're we're we're looking more at how do we improve this day two story? How do we How do we improve people's day two ability to perform actions, backups, restores, and extend upon that? Um and then we have a whole bunch of operators that that are working with this, Kafka, Cassandra, Spark
Um there's there's also a a Kubeflow uh operator that that D2IQ maintains. Um we're checking out. Uh that that that that wraps up a whole bunch of these components as well as the Kubeflow components and uh and distributes them as well. So thanks everyone. I'll I'll be able to switch over for a quick demo showing a deployment of of of Cassandra and Kafka. And then I guess turn it back over to Lexi and or answer any anyone's questions or discussion and then turn it back over. Yeah, thanks Jared. That was very cool overview of a Kubernetes and operators
I guess let's see folks have questions. Um Yeah, in in the meantime, I'm going to do a um quick install of Zookeeper. Um Oops. Um In order to have Kuda as a cube control option, do you first have to install Kuda itself? Is it uh Yeah, uh so so so it's it's a binary that's called um um kudo so they use Git style plugins inside of kudo. So yeah, we um we we distribute um binaries for every platform as long as it's in your path it'll and kudo can find it it'll be installed as a plugin. Nice. So it's just a binary basically like you just need to have binary uh wherever you run cube control and then it will be Exactly, yeah. And and there's there's a um there's a package manager for Kubernetes called crew
Um k r e w that allows you to go find these kudo plugins and and kudo and plenty of other great tools are on that as well. I've put put that into the um the zoom chat. Um Is it similar like homebrew Mac? Is it just like a package manager for Linux getting stuff for you? Uh well no this this is the same as homebrew, yes. This is So so homebrew and Linux brew are the same thing. Um uh it's just a different tap, I guess, in the same repository. Mhm, nice. But I I installed the same way. I went to brew.sh I ran the the curl and everything else
Oh, that's cool. Yeah, so if I get my paws now I have um I'm doing some some cluster API stuff, but uh uh if you look here we have our running Zookeeper instance and now I can go and install Kafka. Um the interesting thing about Kafka here So one one thing that we do uh and very very focus very heavily on is uh this ability to do things either in parallel or serially. Um you know You can't necessarily just run uh This should work once the search generates. So, if you notice here, it started off generating a certificate. If I were to go look at the the the plan status, um Kudo plan status instance equals Kafka instance, we can see here now that we have a uh running deploy plan. Um we're doing a serial deploy. And so, what what serial deploy means is it'll deploy every single instance of Kafka one at a time
Um this may take a few minutes because it's also doing um Yeah, it's letting the the the the TLS certificates all generate. Uh but it's going to it's going to deploy Kafka zero, Kafka one, Kafka two uh all serially and join them into the cluster, right? Cuz that's a that's a manual step with something Kafka like some of the etcd and and whatnot. Um And so, having this ability to have a a a directed acyclic graph of of all your plans and how everything's supposed to like pull together is is really powerful when working with this sort of uh stateful software. Um and then coming to Kudo soon is stuff like support bundles, right? Like not only do do let people ship um the run book with it, but also all the support information with it, right? So, you can say like, "Hey, uh you know, kubectl Kudo diagnose this Cassandra instance, right?" And it'll it'll create a support bundle. It'll tell you try to tell you best of its ability what what it thinks is wrong. Um but better enabling enabling people to uh systematize their run books cuz I cuz I I I I think a lot of organizations uh spend a lot of time rebuilding this sort of knowledge and keeping it tribal or uh otherwise otherwise it's inaccessible. I'm not sure why Kafka's not standing up. I was standing up for me earlier
But hopefully everyone get sort of gets the idea. Um I'm not sure if my kind cluster is messed up from some of the other work I've been doing. doing. So yeah, that's it. That's all I have. I'm I'm happy to answer any questions. I'm happy to engage in as much much discussion as we want to have. Cool
Thank you. I mean, I have a kind of general question. I wonder, you know, if maybe it's too vague, but you know, uh interested in your kind of insights of this. So So um you know, uh I kind of I'm thinking through an example of stateful machine learning application, right? And so the kind of the last few years, there was this whole cottage industry of model deployment, which I first didn't understand why do you really need an industry of model deployment startups, right? And then kind of I realized that essentially like business owners don't know that a model is just a serialized object from a program. Now, we should kind of store version differently from the rest, right? So So they kind of kind of communicate or have like they they explain to managers like here's a model, and then you deploy it, and then you measure some metric how much money it makes. And if the model is bad, it generates like losses, you you quickly roll back, right? So So here's kind of like you know, everybody and their brother are making this model deployment startups and systems. So can I use CUDA to write essentially a model deployment application? So the state is the model, and it has a metric how much money it makes and if it starts making losses instead of making money I need to roll back to a previous model. Like if I wonder if it's kind of it's a good if it's a good example for Kudo operator
Like I want to write a machine learning deployment model deployment operator using Kudo, right? And so my my operator will more you like deploy models and they will be models will have metrics and if metrics show it models are bad Okay, Boris is signaling to me that this is not a good example, but I I wonder I mean Boris feel free to join forces here. Like you know, this is kind of how I see you know, I have a distributed system, right? And I need to send this like model around, right? Because I run I run a web e-commerce app. Like I my model is everywhere, right? So so I just wonder if if this is kind of what what can be used I I think the analogy I'd use is is um you know, it it would it would be like using Kudo to serve your API by having Kudo run a container for you that runs like python -m simple HTTP server, right? Like if there's a tool for the for that model serving. So so so actually back it up a little. It's not like Kudo it would not be a good like pachyderm for example, right? Like could you string it together because you're just creating arbitrary actions on top of um other containers? Yeah, sure. Like there's nothing there's nothing wrong with that. It's not going to s- You're no matter what you're going to have to solve that versioning need in a different way be either storing as a docker container or whatever model tool. Kudo's going to be a better option for orchestrating whatever is is backing it that sort of tooling
Mhm. So in So in terms of Kafka or kind of, you know, like an an upgrade, right? Like everything which is required to upgrade something, and that that would be a good use for Alexey is Zarat was explaining the main idea behind Kubernetes operators is you don't tell how, you tell what. That's declarative functionality is what is in the heart of the Kubernetes. But this relates not to the applications themselves, but the actions on the applications, deployment being the primary action. Sounds about right, Zarat? Yes. So Kudo in this sense is a framework that allows you to simplify building this deployment and life-cycle management operators for the applications. Yeah, it's it's a it's applications themselves. Yeah, it's it's a it's an orchestrator first, right? It provides that orchestration layer that and compared to other orchestrators, it's application aware, right? So so you you you can you can do you know, it's it's thinking in terms of what your application's doing, not just how the topology is laid out
In the fact that you can run commands directly, right? You can you can you can do all sorts of things. Um uh but yeah, it's not a it is not trying to replace any of a tool that you would have covered by an a uh an application suited for that purpose. If you want to uh Alexy and Boris, if you if you wanted to, for example, uh ensure that you deployed Zookeeper first and then Kafka and then Cassandra and then deployed Jupiter matrix was well connected, all the ports lined up, like like everything was connected well together. Um and then you had a plan that was going to uh you know, restore data, like restore your models in into whatever serving application that you had. Yeah, that that that that's that's a great area for Kudo. Um Kudo itself managing the versioning of your models uh another way level of abstraction, I guess, for for what Kudo's really suited for. Got it. No, that makes sense
Thanks, guys. I understand now. It's kind of planned. So, like, what can be planned like, you know, like a graph of actions, that makes sense. Yeah. Uh and so and I mean, like, pedestrian question, like, is there any preference in which like, how can I write Kudo, you know, operators in Kudo? Is it Go-driven? Can I use Scala, for instance? I like Scala better than Go. This is This is all YAML-driven, actually. So, so I really I'm trying to get to this point where we have um where we can do data-driven operators and let people focus on what they want to do
Um so I'm so as far as like what they're good at, right? I I um I There's been a lot of focus around this being as no code as possible. And if you're going to write code, it's on terms of something that you can package as a container, right? So, I don't think Kudo as of right now I can say there are no plans for Kudo to include an SDK that is aligned towards any sort of programming language. Uh cuz yeah, I'm I am I am like a heavy like closurest, right? And so uh uh I'll take closure any day over YAML. Yeah, right? Right? You know, I've actually I've I've been working on like sort of a dynamic framework that uses Eden for um for trying to define workflow steps and everything. So, I'm still playing with that, but uh uh uh I think I think that would be a great great great inclusion, right? You can write um I I I have been playing around with an idea of of if you can compile it down into web assembly, then you can you can add in your own logic, right? But, I think I think the the the dance I'm trying to do or what what we're trying to do with Kudo is not suddenly including a whole bunch of other binaries, right? You want like how do how do you sandbox this properly? How do you allow people to have advanced logic um without tying them into an SDK. And if you're going to do that, why not just use operator framework, right? Um I think in addressing sort of the issues with writing operators in the advanced Kubernetes knowledge and everything else, um uh I think data driven's a good start. And and I'm really want to look at what web assembly and other other tools to allow people to build upon that. Cool
Oh, that's interesting. Any other questions, folks? Uh yeah, Jared. Uh the question that I was dying to ask for the longest period of time. There is Kudo and uh there is the framework with Red Hat that allowed you convert any helm into kubernetes into operator. And they seem to be very very very similar. What are the main differences between the two just bullet points? Yeah, so operator frameworks helm mode still ends up generating out of CRD for that helm chart which is which is fine. It's not where we're going. Um Uh in in the kudo perspective I eventually wanted to the the the pathway for kudo is dynamic CRD schema management
So if I define an operator, it will manage the CRDs for that for you if we even do that. We may stick with instance and operator version for all of time. Uh so so that that's number one. Um Number two is even though you're doing a helm operator, it is still uh uh a a hetero or or mono um there's a one-to-one mapping between operator and the software you're trying to operate. Um whereas kudo is you know N to M. You you can run multiple kudo controllers and control multiple different operator types. It's it's um a polymorphic that's what I'm looking for in in that sense. So so kudo is a polymorphic operator whereas even though you know you you can turn any helm chart into a Red Hat operator framework operator, it's still monomorphic, right? Uh So you're going to be running uh one operator for every type of helm chart you convert
Um that's a huge that's a huge difference. Um and I think it also creates different demands upon Kubernetes and CRDs. Uh the uh those are my main ones at the moment. What do you mean by dynamic CRDs? Because this is one of the largest problems in the Kubernetes because CRDs are freaking global. Uh yeah, it's just a yeah, it's it's a it's a complete mess. It's a mess. Yeah. I I want to Ah
If if if if if this all lands right, I want to work Yeah, of course thanks Sachin um for for joining. Um So uh what I would what what what I want what I want is if you have operators registered with Kudo with the Kudo like um Kudo's declarative language that Kudo will manage creating and the life cycle of the CRDs um for for that particular operator. Um we haven't seen much demand for that, so that's why we've sort of stuck with instances. And also there was a there's a bug that needed to be fixed in CRDs with um uh watchers that wouldn't expire when you If you If you create a CRD, delete it, and recreate it, um there's a bug there was a bug around the watchers. So that's why we didn't do it initially. Uh as far as that watch not expiring properly and then not letting you rewatch. So anyway, ignoring all that, um uh I I I would like for Kudo to get to the point where um it will create the CRDs for your operators for you and then clean them up when there's no more. But also have the abstractions in place to where you just it will manage any sort of dealing with conversion web hooks for you to the best of its ability, I guess
Um still very early, but I I mostly want to try to find a way to like automate away the what I feel is a um CRD system that that honestly needs to be redesigned. So. Maybe not the best answer to your question, but I feel the pain. I I I I I think I feel the the same issues that you're you're speaking to. Oh, close enough. You basically convinced me that there is no hope. Yeah. I mean the the the I I think we need to redo the CRD system
I think I think I think that would be the best for it, right? It just um Yeah, cuz when it comes to versioning and deployment of multiple versions, if they have to have the same CRDs names, everything goes to hell very quickly. Yeah. I I I I think I think ultimately we'll sort of be bound in Kudo by um I I have a very very big interest in what what the next version of CRDs look like. Uh because I think I think it's going to be critical to uh both the growth growth of operators, but also how uh growth and success of operators. But further than that I think I think I think sort of that will like It would be the first start of like undoing the mess that that we we've gotten ourselves into. I I I don't think I don't think a couple years ago when we started creating third-party resources, right? Uh before they became CRDs, people were envisioning or imagining how they look today. And uh And there's there's a lot of issues with with CRDs, the way they're implemented and everything else that makes them very inflexible. So, to the best of the ability, I want I want Kudo to be able to dodge those issues
And then the Kudo team is is is is involved in like uh iterating that. is not this. I think the problem is operators became too popular for their own good. Yeah. Now everybody and their brother is writing its own little operator. So, we get overloaded with this little Manusa operators all over the place. And it's the usual thing for our industry. If there is a tool, I have to freaking use it
I don't know why, but it's important to use it. Oh yeah, seconds on Homebrew. You can install it. Yeah, no, totally. Cool. Well, thank you guys. I think actually um I'll need to run in a few minutes. So, um unless you have other questions, uh it will be a wrap
Um and you know, thank you Jared very much for um sharing all of this. You know, I think it's really cool we can play with this. You know, maybe you kind of you know, it will be interesting to follow up like, you know, maybe uh on the next stage of Kudo, next release, or you know, next release, or what happens. So, we're looking forward to follow this, play with this, and you know, we'll ask you some questions. So, if you guys have some questions, you know, comment under the meetup, you know, we'll get them to Jared. So. Thank you so much, guys. Thanks
Talk to you later. Thanks. Bye.