Scaling Multimodal Data and Reinforcement Learning with Ray | Jaikumar Ganesh, AI By the Bay 2025
So that's me um head of engineering at any scale creators of ray have spent around 20 years working in the Bay Area in various um tech startups and companies. Um started my career in the early days of Android. spent some time uh building the ML platform at Uber and I've been at any scale for some time now. So the topic today is scaling multimodal data and reinforcement learning with Ray. How many folks here are familiar with Ray? All right, three hands, four hands went up. All right, cool. So I have three takeaways for you. This talk is going to be like a Costco value pack full of insights
Damn, that joke didn't land. Okay. Um all right, so let's get started here. So you know each computing era um was built on a new compute stack right you in the last 25 years we started with um client server architecture you had your CPU servers Oracle Unix um era and then as we moved to internet and web services virtual machines came in and then you had the LAMP stack Linux Apache MySQL PHP fast forward then you had your big data era, microservices, kubernetes and containers, radius, spark, etc. So at a very high level, this is how the technology stack has evolved in the last 25 years. But of course, we have now entered the AI era. What does that mean? The core hypothesis here is that the AI era needs its own computing stack. So let's try to understand why there's specialized heterogeneous computing hardware now required
You see all the chip vendors, the cloud providers having their own AI accelerators. You see TPU chips. You see all all the news in Nvidia about Nvidia and AMD creating new chips and new AI data centers have been created specialized data centers for training these large models and they have different networking stacks that's required. So fundamentally the computing system the computing hardware that is actually required with GPUs and with all these specialized accelerators has changed in the last 25 years for AI. The second thing is that you have different stages of an ML pipeline. You have your data processing stage, your training stage and your serving stage. Right? Generally data processing is done on CPU. Training is done on GPU and LM inferencing is done on GPUs too
But that's now fundamentally changing too. Each stage is becoming a heterogenous compute system. CPUs plus GPUs. What do I mean by that? What do I mean by multimodel data processing? Right? It's no longer text to data. We are now talking about images, audio, video, self-driving cars including this talk can get transcribed. You have your tools which are doing meeting note summarization right and I'll go go into more detail into this aspect that has become CPUs plus GPUs from pre-training we have gone into post- training era and re reinforcement learning era that also requires CPU plus GPUs and then now agents are coming think about all the coding agents your cursors your codev etc there it's no longer just like oh one model and you're like your results are out it's a continuous loop your model producing data and then you someone changing the code base. It's a continuous loop that's actually happening. So there's a fundamental change that's actually happened in the processing pipeline to move to a CPU plus GPU architecture
Obviously models are getting much bigger. They are processing like you know billions of parameters. So they require a completely different uh processing mechanism. And there's another fundamental change that's happening. Your software systems initially were all deterministic. You had a micros service, you had a database, a model, and you could put it in production. But now it's no longer that because your models, reasoning models specifically are stoastic in nature. What do I mean by that? There's a continuous loop between model, data, and code
Model produces data. That data leads to improved code, newer model, and there's a continuous iterative loop. Right? Classic example. Well, if any of you have used any of the coding agents tool, it's a continuous loop. So, we have worked with various companies Uber, Pinterest, Apple, Shopify, Netflix, etc. And there's a clear pattern that's actually emerging in the AI infrastruct. You have all the AI workloads at the top, your post- training, multimodel, data processing, agentic AI. You have your CPUs and GPUs at the bottom
In the middle is the AI infra software stack. So I don't know if people can see it because of the lighting here but there is lots of challenges in this AI inframp compute stack. You have to handle mix CPU GPU compute that I talked about. You have to handle spot instances because the GPUs are very expensive. You have to handle dependencies because Python programming is a dependency hell if many of you are already familiar with that. You have to handle hardware failures. You have to handle resources. All of this is a lot of complexity
So we are seeing three layers within this AI infrastruct develop. One is your most popular framework for training and inference that's pytorch. Jax has also become a bit popular um with um specifically for TPUs. And then you have various transformer specific models. These are the specialized LLM inference providers. You have your VLM open source and they need this specialized techniques like continuous batching again because you are if you have interacted with an LLM like chat GPT you know that it's not like a oneshot thing you it's a continuous loop of tokens that's actually going on so there are specialized techniques to handle that and then these as these models have become bigger you have various spe parallelization techniques that are also needed like you have to have data parallel model parallel etc so there are specialized libraries for In the distributed computing era, Ray is dominating a lot over here. For the people who are already familiar with Ray, this should not be a surprise. But Ray is powering Chad GPT
Ray is powering the ETA models that you see at Uber. Ray is powering the recommendation algorithms that uh Netflix has. So what Ray provides here is finer grain scheduling, fall tolerance. So when you're especially running these on these heterogeneous nodes, you need to make sure your code can easily recover from failures of the hardware of um the software and then all of this is built on Kubernetes. Um slurm is also a job manager that is uh picking up some steam over there. So Kubernetes is responsible for bringing up the pods, bringing up your containers and now finer grain scheduling within that container is what the rail layer um actually handles. All right. So hopefully all of this made sense especially for folks who haven't spent much time in this area
And this brings me to your first takeaway. There's a transition from the LAMP stack to the park stack. What do what do I mean by the park stack? Pyarchch AI foundation models and specialized frameworks ray and kubernetes. This term was uh coined by Ben Laurica and Linux foundation has also picked up this term. In fact, this term is resonating. It was just coined two weeks back when Ray was donated to PyTorch. So, if this term picks up, you'll be among the probably among first 200 people ever to realize this. Um, but the concept is still there
Pyarch, all the AI agents, Ray and Kubernetes. So, this is what the Ray ecosystem looks like. You have your ray core. Ray core is responsible for all the scheduling. You have specialized libraries around data serve train reinforcement learning and then we have a rich ecosystem that is there for various libraries that are created on top of Ray. So the first question is what's the best data processing framework and architecture for multimodal data. As I mentioned multimodel data has picked up a lot of steam right images you see vos and Teslas uh going around. Um there are automated trucks now
There are like you know bedrock robotics if you have heard of it is uh automating the construction industry. So what's the best processing framework for this? In the olden times or probably not olden times in the last 10 years data is like you know was mostly structured in tables. Bottlenecks were CPUs and GPUs. You know you had your ETL pipelines, data frames, you do your shuffle operations, write it back to the database and ecosystem was basically Java and Scala based. These requirements have changed. Now you need to deal with tensors, not just tables. You need to deal with CPUs, GPUs in memory. Your operators are now heavy and they're distributed across a cluster of nodes and the AI ecosystem is in Python
So whatever is the data processing framework that's used for your ETL pipelines cannot really work well in an AI world. So ray data is a data processing engine built for the AI workloads. Firstly, there's support for multimodal data. You have support for um readers for images, video, audio. The back end is based on Pyro and NumPy. Robust support for accelerators, GPU, TPUs, optimized GPU feeds, things that I talked about that the AI infrastructure has changed. You have various stateful operators. Sometimes these expensive GPU states needs uh to be materialized
You need to reduce the memory pressure especially in a distributed um system. And then it's a AI first ecosystem. It's integrated with PyTorch, TensorFlow, VLM and SGA. The fourth point is extremely extremely critical because at this layer what you do is not just take one building block and serve it to the users. You have to build it. You have to build it with your training ecosystem. The park stack that I talked about. You have to make sure the entire stack is interoperable
So whatever your data processing framework is should be well integrated with PyTorch, TensorFlow, VLM, etc. So let's take a example. Now you have a computer vision applications. You're reading a bunch of images from your S3 bucket. You're resizing them. You're segmenting them. You're classifying them. And you're writing it back
This is what it looks like in a heterogenous world. your resizing operation and reading operations happens on the CPU. You have two inferencing operations. One is the segmentation model that's happening on a GPU and the classification model that's happening on a GPU and then you write images back to the CPU. Many times you get these GPUs and your finance team is asking, hey, why are my GPU so underutilized? The reason is that your CPU is still busy resizing the images. So the beauty of ray data is that once an image has been resized because data can be streamed to the memory immediately sent to the GPU to keep it busy. The second thing is that each of these models can be independently controlled. You can specify the concurrency for the segmentation model different from that of the classification model
You can specify the number of GPUs for this model different from that. And so each stage of it you can control. And this is where the power comes in. This is what I mean by heterogenous data. This is what I mean by controlling every single stage to increase your throughput, reduce latency for your users. It's a classic computer vision application and also improve your GPU utilization. So some of the architectural advantages streaming execution which I just talked about the CPU to GPU as the resize image immediately flows and since it's built on top of ray core if there's a failure in any one of the stage it will automatically reconstruct it back higher GPU utilization and there is concept called object store um which allows easy way to move these objects in memory like you know you have a zero copy on a single node and you know it's efficiently in cross node data transfer right this is a concept that's built into ray code so some success stories Amazon uses ray data a lot they have reduced latency by 90%. Roblox uses them
Pinterest uses ray data. Cursor uses ray data in their model. If you're actually using cursor, cursor races uses ray data. Reflection AI native startup uses ray data. So there is bunch of use cases of ray data. Real use cases, people using it at scale both for digital natives as well as um AI natives. So that's my takeaway number two. Ray data is an excellent framework for multimodel data
Hopefully I explained why that's the case. Now let's talk about RL and post- training. This is the new hot area. So RL and post- training. So you had pre-training. What is pre-training? You have this internet full scan it. You develop a model and then you have a model of the world at least what is there on the internet and your model can serve questions answers on it. Post-raining is about taking this pre-trained model and making it work really well for a use case
So the chat GPT model is post-trained model. What cursor uses is a post-trained model. So what does architecture for it look like? You have a trainer that's the pink pink box over there. And then the trainer is generating the weights and then you also have a generator. Generator is the yellow box. you have to do an a continuous roll out of it something like using VLM and there's a continuous loop that exists between the pink and the yellow boxes where the weights need to be transferred it's also integrated in an environment the tools environment over there so you have an agent let's take a coding agent right the user is typing checking out repositories accepting changes interacting with the environment that's what the environment comes in and then you have a mod module which is deciding which of these options I should pick where do I actually maximize my reward that's what the observation reward is. So this whole thing increases the complexity a lot more. Now weights needs to be transferred between CPUs and GPUs and you have to interact with the environment and that's where Ray actually shines
So let me explain why. So you have two classes a training step and a generator step. A trainer is training the model generating is rolling it out and generating data. They need to exchange weights between the two of them. So the way ray does is very very simple. You have two annotations you have to boot ray. Ray dot remote and you can specify any compute frameworks. You can specify you know number of GPUs etc
And all you have to do is trainer remote generator. So these two have become like remote actors now that can run on a distributed computing cluster. Is this simple to get started with Ray and build up a distributed application that is across running across multiple nodes in your cluster. But for RL, you need to go another step. You have a while loop. You need to train. You need to synchronize weights and you need to do rollouts. So you call trainer.train.reote to get it started
Generator generate remote and you exchange weights using the two functions that is created. But there's a problem here. This will work but it's inefficient because now you're transferring the weights between the trainer and generator through CPU through GPU and so we have also added something called GPU object transfer over RDMA so you can actually specify the transport back end that you need to have so that it directly transfers people who are familiar in this area would automatically get this but for the people who have not worked in RL or in this area the takeaway here is that ray makes easy for RL reinforcement learning examples and makes it efficient by allowing these easy ways and efficient ways to transfer from CPU to GPU. Again, comes back to the first point I actually made that the AI era needs a new computing stack. Here's another example why that's the case. Nearly every open source RL framework library is built on Ray. around here around 10 of them are listed eight of them are built on ray and and some of the popular ones like cursor etc uh are also built using ray so that's my takeaway number three ray is an excellent framework for post training so how will you build your AI stack you're thinking about all these things yeah that's great I understand but how do I build my AI stack in my company you have two options or three options you extend your existing stack whatever you're using for non-AI use cases to build AI you build on top of the frontier models or you create a AI native infrastructure stack so extending your stack will fail because you need these finer grain scheduling for some of the examples that I actually talked about your GPUs are going to be underutilized and you're going to be failing to scale for your future AI workloads right multimodel data tomorrow something else new will come and so you know your your stack will actually fail to uh live up to that expectation. It's the classic innovators dilemma
You have something that is working for your use case. These new use cases are coming. The easiest way is like oh let me just extend works for some time but then you'll run into all the problems that's actually mentioned. Second thing is yes you can build up on top of frontier models right but you will lose some power. If you're a small startup five people I'll totally say do this but if you're past that you will lose some control. you have to send the data u to those companies you will not be able to fine-tune or really tune the way um uh I talked about your agentic systems can be built but then you your complexity increases and you really want a lot more control over the pipeline and the most important aspect you need to build deep in-house AI expertise the engineers here will totally understand it maybe the execs may not because it's all about finance and budgets but you need to build deep [music] AI expertise and this takes time, right? This takes time. So what's the answer? Raise the answer. But you don't have to believe my words, right? I will give you ex proof that lot of companies are actually building on top of RA, right? For example, this is Ray's adoption
Um, so if I'm not sure if people at the back can see it because of the window lighting here, but the number of downloads per week has increased in the last one year from 1.5 million to 7 million. Ray has been in existence for some time. But why has this increased? Because multimodal data workloads has increased a lot. Reinforcement learning has increased a lot. So people are actually seeing hey Ray is a great framework for betting on for making their infrastructure stack. Examples of some of the companies which are actually using Ray today you have Cruz, you have cursor, Uber, Door Dash, Riot Games, New Bank, Adobe, Canva, Roblox. All these people are actually using Ray in production. But that's not enough
You do pip install ray fine you can run all the things I talked about in your laptop but holy if you have to run them in production. Okay you have to manage the compute you have to scale the compute. You have to do monitoring. You have to do observability. Your developers will say I have this weird Python dependency. Please take care of it. Uh and you have to handle dependency management for them. You need an easy way to scale your jobs and compute platform
That's why we actually built any scale. Any scale is the managed version of Ray. It provides multi cloud experience. If you have AWS, GCP, some NeoCloud like Lambda Labs, any scale can efficiently find the GPUs. It can do data lineage tracking etc. So if you have a huge infrastructure team, then you'll be able to manage Ray by yourself, but if you don't, and not that they're sitting idle, so and if you don't, any scale is an excellent choice for um uh building Ray. again a bunch of uh customers who are using any scale for the reasons I mentioned. I do want to call out a couple of them today
Um Pronovo is an AI first preventative healthcare uh company. What they do is a full body MRIs and they are able to figure out anomalies biomarker correlation. they had trouble with their pipelines and they used all the three layers that I showed in the previous slide and they were able to do 5x faster development and 40% reduction uh in cost and they were also able to run this at pabyte scale a multimodal data processing pipeline and at the same time able to meet the security and compliance needs um of the healthcare industry. Geotab is another example. It's a AI fleet management company. They collect telematics data, multimodal image processing data and they also used all the three layers of the any scale stack along with ray. They were able to get 43x video processing throughput improvement and 40% less GPUs needed for peak. There are a lot more examples
You can check it out on ray and any scale website. But hopefully these two examples gave you a little bit of a taste of real customers in production doing multimodel data and what is the impact they're actually having. All right, three takeaways. Let me refresh you all with the three takeaways. Park stack pyarch AI frameworks and agents and frontier models ray and kubernetes. This is gaining tremendous adoption. This is a production ready way to think about your AI infrastructure to build your agents, to build your multimodal data processing pipelines, to build your RL stuff, to build your training, to build your serving, any number of a IML use cases can be built on top of this. And if you're in building some agents and you want to integrate with say lang chain or langraph, all of that is totally possible because all of this is in Python
It's a Python native ecosystem. My takeaway two, ray data for multimodel data processing. Hopefully, I've given you examples and explained some of the architectural reasons as to why this is a great framework for multimodel data processing. And ray for post-raining. Post-raining is a real workload that's just going to keep increasing. If you are in the agentic world, you have to work with post training and ray is a great framework for doing that. All right, that's basically what I had. Three takeaways
If you want to learn more, docs.ra.io ray dos everything is open source. If you want to try any scale, go to console.nyscale.com. And I failed to generate a QR code for LinkedIn. So I just basically put this uh linkedin.com/jikmar Ganesh and you will actually find me right [applause] cool. Awesome. So we have five minutes I believe for Q&A. >> I have two questions. Um, one is fractional fractional GPUs
Why is that important? Um, and like given the hunger for GPU, is anyone actually care does anyone care for fractional GPUs? >> Yeah. >> And my second question is um what are the agentic frameworks that are tailor made for Ray? >> Yeah, good question. So fractional GPUs I think there's there's a difference that we have to always think about because sometimes when it comes to GPUs we think about like oh Nvidia is producing all these GPUs u a they're expensive b a availability is still scarce and c is like you know what we hear is from the openis and the metas of the world right if you're working there sure you may have the money to buy a cluster of GPUs but when you're not working there you uh you have to make sure your utilization of the GPUs there you may not need a full GPUs um eight cross machine you may not need a full all the eight GPUs in the node you may just need a one GPU and you can use the second GPU for something else right so the fractional GPU support is extremely useful even at openai and meta etc they use fractional GPUs for their workloads it's all about increasing utilization because they want to really get make it more and more efficient Um the second part is which are the agentic frameworks which work well with ray. Like I said ray is um python based python first. It integrates with vlm of agents use vlm and all the agentic frameworks like lang chain lang graph. We have written detailed blog posts uh on how to use lang chain and lang graph with re. So for example you have a document you can chunk it using ray data and then you can have the entire workflow done with lang graph and you can use um lang chain so all the frameworks that you actually have will work well with ray ray is the compute substrate agents are the applications above >> any other questions all right I'll ask the question again so I think four hands went up when I asked How many of you are familiar with Ray? Now I'm going to ask the question again. How many of you are familiar with Ray? All right, few more hands up
How many of you got at least a reasonable understanding of Ray that you can go talk to teams like, "Hey, we should look at Ry." All right, there you go. These are your resources. Um, and I'm not sure if this talk is going to be recorded and put up. >> Yes. Uh, will it be published? I know we are recording for sure, but will the talk be published? All right, I I'll find out for you. >> All right, cool. So, uh there's lot of material on YouTube, lot of uh material on the uh web and you know, we just did our race summit two weeks back. The keynotes are and the uh on YouTube there are detailed speaker sess speakers from um various companies
All of that is publicly available and if you have any questions feel free to contact me, right? Thank you so much. [applause]