LLM Avalanche: William A. Gaviria Rojas: LLMs in Production

LLM Avalanche: William A. Gaviria Rojas: LLMs in Production

Recording: LLM Avalanche: William A. Gaviria Rojas: LLMs in Production

thank you good afternoon everyone my name is will I'm one of the co-founders of co-active AI uh this is one of the I think I'm the first Speaker for the open mic session so thank you all for coming uh so okay can you guys hear me now is that better okay awesome okay all right let me know if I'm not speaking loud enough first please hey so everyone my name is will gaviria I'm one of the co-founders of Kauai uh today I just want to tell you a little bit about some of the lessons that we've learned we're building data plus AI systems echoactive uh thank you all if you want to connect with me I'm going to be hanging out here so would love to just chat all right so one of the things I want to start off is just talking about data one of the things that we see is that data has just been evolving over time overall we're going from you know 10 years ago we thought of data just being these tables now and more and more we're thinking about being content right is text it is images it is video it is a combination of all of these um so data is changing right and so so have the tools that we have to make sense with them and that's where data that's where AI really comes in so to kind of quote Bill Gates right he said content is King and if content is the king the king is already here right 80 of all structure or 80 of data is already on structure is going to be unstructured by 2025. I think really the important thing is regenerative AI we uh we add so you're about to get an explosion about structured content because it's just so it's going to be so easy to just make a lot of this content and so uh we're just going to generate a lot of it over the coming years and we have to make sense of it and so that's where AI really comes in so that's truly the data plus AI data is essentially the bread of all of this and AI is going to be what we're going to need to use to make sense so what's the point though right like if data is okay sorry if content is the king and AI is a queen right it's a royal marriage it's actually not very easy so this is what the stack looks like today for a lot of folks you have a lot of data being generated some of its structures some of it's semi-structured so it is unstructure for the structure and semi-structured data there is a Playbook that already exists right I mean there's a lot of work that we did to get there but you can kind of make sense of it for unstructured data though it's a little bit of a mess we store it we archive in a blob store and we don't do a whole lot with it afterwards or at least most companies don't and so what I want to talk about today is you know what happens after we capture all this instruction content and that's where it gets pretty hard a lot of folks have a lot of difficulty doing stuff so they actually end up doing nothing that's the easiest path a lot of folks hire human labelers that's what happens a lot with images a lot of folks have ai power apis that's where a lot of us are here today frankly right but I want to talk about moving towards reliable scalable adaptable data systems that are AI power and that's absolutely what we've been building on a coat there for the three plus years uh and there's a few lessons that we learned along the way that I just want to share with you because I know a lot of folks are in the do nothing station they're getting started and so I just don't want you all to make the same mistakes that we see over and over again from everybody in the field the first thing I just want to set the tone despite everything that happens on the twitterverse and what you see folks talking about the reality is that most Enterprises are just not doing AI right AI adoption is growing rapidly but most folks are still aren't using it so if we're building tools today for these folks we have to empathize where they're at and meet them at this stage right they're just getting started with AI how do we build these data systems for them and I think that's like that's a key part of just empathizing again with the customer and something to keep in mind as we build these tools for them the other two lessons I want to go over though are actually key if you're one of these folks who are designing these Data Systems so the first lesson is uh logical data models matter a lot more than you think I'm going to go over these slides mostly for like as a visual reference but feel free to just ping me afterwards if you want the slides I'm just going to go through them a little bit fast but really the key is this a lot of folks see AI is this kind of monolith that sees between the data that comes out of the model that people use and the data that sits in some sort of blob store like a key Value Store most of the time and the pitfall that a lot of people have is that there's a clear owner of The Blob store the data engineering team type of thing there's a clear owner of these Foundation models that people leverage like the AI team the ml team but this handoff is often overlooked and what ends up happening is that this is actually a huge technical depth in Disguise because when we store these things we think about them for a standpoint of a key value store right some Json and there's some bits and bytes assigned to it but it turns out that there's AI models that's where you have a lot of specificity it is Data specific right is it text is it audio is it image this is a video it is also task is specific are you doing object detection are you doing sentiment analysis are you doing captioning and this combinatorious of the two absolutely leads to quite a lot of variety in The Logical data model of the inputs for these AI models so what do we see happening time and time again no one actually owns this impedance mismatch the data team builds something that doesn't really work for the AI team the AI team built something bespoke that doesn't really scale and the end result is that all the uai solutions are ultimately bottlenecked and no one knows why I just want to go a little bit more in depth into this into the technical depth that happens if you don't think about it because if you have some piece of structure text like this and it's going to say some summarization task that you know maybe you don't need a transform and this is a really simple impedance mismatch there is no mismatch so just feed what's in a key Value Store into the AI models but if you have something slightly more complicated like a language detection task or a sentiment analysis task you may need to do some transformation you may need to do some key phase distraction but the whole point is that what goes into the AI model no longer looks like what store and actually capturing this transformation is super key in unlocking the next level of of quite frankly scale because what we see when we see success is folks that then think deeply about this they get together with their data folks and the AI team they build a hybrid team they build a lot of impedance mitchman transforms they build a lot of different logical data models and then all of a sudden your Solutions actually can perform pretty well and can scale um not only does this end up resolving the AI bottleneck but you also get a lot of Pathways for optimization one of the things I want to talk about is for image pre-processing when we saw this is I'll give you this is a real example of something we saw happening where you have over here three AI researchers are feeding images into the pytorch models they're all fetching the same thing from the key value store and image they're then doing the same transform and then they're using this images and feeding them to their models but if you see this this is insane because they're all doing the exact same compute the exact same i o three times so if you think about this a very simple model is simply fetch it once Transformer wants serve the the smaller transform image and then what you end up doing is you reduce compute you reduce IO and you actually get higher GPU efficiency because you're spending less time on i o and doing this computation that is frankly not valuable for an example of AI so this is just one example of many examples another thing also I wish I had more time to talk about is you can actually leverage a lot of the outputs from these Foundation models to Cache compute right unfortunately I don't have time to enough to talk about this today but come see me we'll love to chat some more we have an awesome follow-up talk from Sam from redis who's going to talk a little bit more about this so big shout out uh and then some important thoughts I think for a lot of this for a lot of a lot of us are building these tools now I think the ones that are really going to capture the opportunity are those that build tools that actually scale data pipelines and infrastructure it seems kind of boring but this stuff actually really matters so if you find yourself planning for something like this but your solution actually ends up looking something like this that just feels really finicky and and feels really fragile that you might want to take a look at your foundation of what you're doing even before you get to the AI Park I just want to call one of my buddies Joe Reese who said it really well because I do want to empathize where you folks are building this because sometimes we're just given an impossible task because the reality is a lot of this data is really in shambles right at the Enterprise level it's expecting to stick llms on top of an infrastructure that's not there to support it it's just a very difficult problem so just encourage you all to you know talk to your fellow data Engineers talk to your family AI get together in the same room and you can actually solve a lot of these problems and build AI Solutions at scale the importance of scale though I just want to give an illustration is that you know we're moving from data Lakes to Oceans right if you think about it for a tabular data source let's say you have a bunch of numbers and you have 10 million rows that's going to be about 40 megabytes right just to give an illustration let's say that's the size of Lake Tahoe right when you switch from say flow 32s to text you're going to make a huge order of magnet to jump you're now not in 40 megabytes right in 40 gigabytes and now you're going from Lake Tahoe to something like the Caspian Sea which is the largest lake in the world the second you go from text to images this gets even worse and you're now talking about the terabyte regime and it's like you're trying to cross the Pacific Ocean and this canoe you made for Lake Tahoe so that's really what we need to build this tool is that truly actually capture the i o we're seeing some of the challenges of building these things today due to memory for proximation in Ram and GPU for uh for text but the second we go to images and video IO Network latency all these things even this space becomes a huge limiting factor so again I just want uh just motivate that you know scale matters a lot logical data models uh logical data models matter quite a bit and if you're interested in this you know this is what we do at co-active we work on these large AI problems at scale uh for images of video uh we're looking for folks to join if you're also thinking about tackling the image of video data you know we think we have one of the best class Solutions so we'd love to connect would love to chat with you just uh come find me my name is will thank you all for your time