Rust AI & Data Meetup: Linghua Jin, CocoIndex
Recording: Rust AI & Data Meetup: Linghua Jin, CocoIndex
I'm glad to be here and yeah, I have a lot of friends in the ecosystem. Coco Index builds with everybody in the ecosystem. I think we don't we're not trying to replace any existing compute with CVM. We really think we're building a kind of compute from scratch with Rust and Oh, okay. Uh so, yeah, we're we're really trying to build a kind of compute from scratch with Rust and I just want to uh walk through some of the philosophies and why we think a new kind of computing infrastructure should exist. So, yeah, like everybody knows like agents runs 50 times faster. I think everybody here today wear Rust for AI and we're trying to build something that is for the agent speed. Uh so, um agent is part of the data loop
So, agent reading like massive like code, traces, plans, decisions and they also they generate like code, like Slack messages, everything, right? And how do you like continuously surfacing a effective data view for the agents while they are continuously creating the data? So, Coco Index tries to solve the gap by creating you can imagine it like a kind of a materialized the view on top of dynamic unstructured data. So, view has all all always been part of the decision process. Like we come from like long time ago web has a server-side rendering similar like batch pipeline. And like when people are sick of like waiting in front of browser and waiting for the page to load there's like incremental DOM update, right? Like jQuery and then there's different kind of ways to implement imperative incremental data pipeline. And then when you have a lot of state, like agents are generating a lot of state and they read a lot of state. So, how do you dynamically surfacing the right view for the agents? There's like React and then Cocoa Index tries to propose this kind of persistent state-driven data pipeline that is declarative and incremental by default. So, yeah, like from high-level it is simple like your your target should always be a function of your source. So, it's a derived of the transformation and your source
And when your source change, your target should be changed and it's reflecting everything. So, this is high-level this is how Cocoa Index works. It takes from like database, meeting notes, these are like not traditional like data processing pipeline that takes and web, file system, message queues, and a lot of like images, stuff like that. And you perform transformation with Cocoa Index and you dump it into a relational database, data warehouse, graph DB, and stuff like that. And the control plane itself will take care of like version, lineage, everything. Um, all the heavy lifting stuff. So, the kind of a data pipeline that Cocoa Index tries to build, like kind of data we build for the AI are like search indexes, knowledge graphs. Those pipelines usually like with these unstructured data you you you you normally have like high fan out during the processing and usually means like multi-stage pipelines
For example, if you want to do like do the like entity resolution for graphs, these are like you need a multiple stages. These are kind of the shapes that traditional data pipeline don't exist, but AI will need to access this kind of data effectively. So, yeah, like if you want to build yourself, there's like a lot of things you have to do it. That's why like we think we want to build a kind of compute that covers all of it. So, which will save you a lot of time. Uh, so Rust has all like Cocoa Index Engine has always been Rust. Tradi- traditionally, like, no, from beginning we have like this Rust core that provides incremental processing, memorization, stuff like that. And we have like Python SDK on top of it with like Rust binding
We're launching a Rust SDK because we're noticing everybody is starting to write in Rust. There's no reasons to like do a Python on top of Rust, stuff like that. So, we're launching like a Rust SDK and from a high level, like, this is how it works. So, so, you begin with a like a app declaration. Um, so, you do like mount tape mount target, which will basically the pipeline will manage the schema for you. And when you have new column like and stuff like that, the day- the pipeline will take care of the schema will evolution for you. And then from there you can declare, like, I'm I'm going to um process a few like components. So, here we define that this one pro- processing component for profile
And then, like, within a file processing you can do like, "Okay, I'm going to This example will do a simple embedding right." And then you just declare your target state. And once you do all of this, the engine will take care of like the incremental processing. So, whenever you have source or your code base is updated, it will do the minimal processing for you. So, yeah, like, it works with your stack. You can take sources from like local files, code bases, EG, you know, OCI object storage. You perform any kind of transformation and these beyond files, uh, you can do like clustering, stuff like that. And then you have like we support different kind of targets. You can also like easy to plug in your own targets and stuff like that
Uh, yeah, I think that's like pretty much it. And we came from like, um, Google on web search team. My co-founder was one of the founding member of, uh, Google's state-of-the-art um, index infrastructure that uh, that was processing at the scale of the entire web. Uh yeah, these are actually some examples you can build. Like for example, if you want if you want to do some code-based indexing stuff, it's like beyond embedding, you can do like mini maps, concept maps, you know, beyond vectorization and stuff like that. So, um that's all I have. Uh yeah. Thanks a lot.