Rust AI & Data Meetup: Emil Sadek, Universal Data Connectivity for Agentic Systems
Recording: Rust AI & Data Meetup: Emil Sadek, Universal Data Connectivity for Agentic Systems
All right, hello everyone. I'm Emil. I work at a startup called Columner and we're working on the next evolution of database connectivity technology. Um so, the main technology that we are contributing to and building on is ADBC. This stands for Arrow Database Connectivity. Um if you're familiar with like JDBC, ODBC, Python DB API, those are all kind of old connectivity standards from the 90s that everyone is still using. Um ADBC is the modern alternative. It's a set of APIs and libraries for Arrow native database access
And it's a subproject of the Apache Arrow project. So, it is uh you know, fully open source, not controlled by one single company. Um Oh. Just some quick terminology. We have like the ADBC API and that's kind of a set of standard API uh the standard API implemented in multiple programming languages that you can use to interact uh with different databases. Um the drivers are what actually kind of connect to databases in between your application and whatever database you're connecting to. A lot of these drivers are implemented in Rust and then compiled into shared libraries. So, they can be used with uh all kinds of different programming languages
Um and then lastly we have the driver manager. That's the kind of language specific library for Rust or Python or whatever language which will load a driver and then connect to a database. And so, kind of at a high level, you will just submit a SQL query to whatever database you want using a unified API, and you'll get the results back in Apache Arrow format. Now, so ADBC is very fast. This is a benchmark that DuckDB did. I believe this is a 38x improvement, and this is not uncommon, especially when connecting to like columnar analytical databases like ClickHouse, Snowflake, DuckDB, Databricks. We often see like 20, 30x improvements compared to like legacy connectors. And this is especially important now in the AI age
So, for a long time, right, we have humans using these slow connectors, and the humans thinking, you know, what do I need to figure out? How do I write this SQL query? Then you submit the query, your connector's slow, get the results back, do your slow, you know, reasoning, and when we have a fast connector like ADBC, the results are coming back, you know, much faster, but we still have this slow human reasoning step. But now with AI agents, this changes completely. And so, the agents are able to reason super quickly, and so when we have, you know, AI agents using a fast database connector like ADBC, we have a really quick time to like insights and value, right? The agents can super quickly come up with like SQL queries they want to run, they can submit those queries, get the results back super quickly, and like keep looping and iterating. So, at Columnar, we've built some tools to make it easy to use ADBC. The first is a command-line tool called DBC. This is for installing and managing ADBC drivers. So, just like you can cargo install packages in Rust, you can do, you know, DBC install DuckDB or ClickHouse, Databricks, you know, whatever driver you want is just a single command and it'll install the driver on your system. And then you can use it in, you know, Rust or whatever programming language you want
Um another tool we built is Databoat. This is a command line tool that will use those ADBC drivers to connect to any database. So, this is a kind of database agnostic tool. You know, you've probably used like PSQL for Postgres or like, you know, Sequelize and like all these databases have these their own like CLIs and this is kind of a unified CLI that you could use for any database. And so, we all know that coding agents love to use CLIs, right? And so, we have put together some uh skills for you to use. So, we have kind of general skills for ADBC. If you're developing applications or agents and you want to use ADBC to connect to databases, we have uh skills that your coding agents can use for that. And then we also have skills for the DBC and Databoat uh command line tools
So, you can use Claude Code or Codex or Open Code or whatever agent you want and it can just, you know, connect to your database using Databoat and install the drivers with DBC and it'll just, you know, run whatever analytics you want. And this works really well. You can tell it, "Hey, I want to connect to my Snowflake and, you know, explore my data, you know, iterate, tell me what I have." And it's super super fast and you can use, you know, whatever agent environment you want. Um that's pretty much all I have for now. Uh here are some like useful resources if you're interested in learning more. You might want to take a photo of the slide. We have kind of official documentation for ADBC, documentation for the DBC and databoat command line tools. We also have a quick starts repo
So, whatever programming language you want and whatever database you want, we probably have an example for each of those. We have a cookbook website which is a little more like long-form example tutorials using ADBC. And then we have a blog as well if you want to keep up to date with what we're working on. And yeah, any questions? >> [applause] >> That was great. Super interesting. The um Is there a setup on the database side as well or can you just do everything kind of on the client side? Cuz it looks like it's an abstraction over any store, analytical usually, right? So, can I pull it off entirely client side or do I need to go change things in the back end? >> Yeah, that's a good question. So, like here. So, for you know, the driver, the individual like database driver is what is communicating with the database
So, you know, for example, the ADBC driver for Postgres is going to like implement the Postgres wire protocol. You're kind of speaking to the driver with the ADBC API. But then behind the scenes, the driver will use like the Postgres wire protocol or you know, whatever other wire protocol the you know, database you're connecting to is using. But yeah, on top of that, you just are using the ADBC API. Some databases actually like will return data in arrow format, which makes things really easy because ADBC is returning the data the data in arrow format, but for other databases we have to do kind of more converting the data within the driver. Cool. Thank you.