Scale By the Bay 2023 intro video by Jim Dowling

Scale By the Bay 2023 intro video by Jim Dowling

Recording: Scale By the Bay 2023 intro video by Jim Dowling

hi my name is Jim Dowling and I'm CEO of hopsworks hopsworks is a feature store for machine learning that is it's a platform that manages your data for training machine learning models but also for making predictions of machine learning models that is inference batch inference online inference one of the challenges you have when you're managing your data for AI is that data is often stored in object stores in Lake has file formats like Apache hoodie or Delta lake or Apache Iceberg and how do you get that data from the lake house as fast as possible to the python environments where we train our models but also where we don't do much of our inference as well so in this talk I'm going to talk about some technology we use based on a database called docdb and also based on a protocol called Arrow flight service as a way of moving the data as fast as possible from the lake house to the python environments where we'll do our training predictions of models some of the things we look at are using the arrow protocol so arrow is an in-memory column format but it's also over the network protocol to reduce the amount of serialization and deserialization of data and also we're going to look at how we can move our data from a calendar format in the lake house directly into account the format in Python clients such as pandas so I hope you can join me on this talk we're going to look at performance and scale and AI all really interesting topics [Music] [Applause]