Scale By The Bay 2019: Michael Paul Armbrust Interview
Recording: Scale By The Bay 2019: Michael Paul Armbrust Interview
[Music] yeah my name is Michael Armbrust I work a data bricks and I'm a principal software engineer there [Music] yeah it actually is my first time here at this particular conference but I heard about you through my one of my co-workers Mattei who's actually the creator of batchi spark he has been here a couple of times and he said it was an awesome opportunity to talk to the community [Music] yeah so my favorite part about my talk is when we get past all of the distractions that are kind of inherent in the old way of doing things where people are just manually storing a bunch of files out in some file system and calling it a data Lake and we get into how people start talking about the quality of data and how they've reason about incrementally improving it until it's ready for consumption yeah that's a good question I think you know for me while data bricks also uses Delta Lake internally for own pipelines what actually gets me excited is helping our customers scale they are storing massive amounts of data in there to their data lakes and they're trying to get value out of them and what Delta Lake does is we actually take that metadata problem and turn it into a data problem and we use spark to process that metadata in parallel so even if you have a data Lake that has hundreds of millions of files in it you can still query it interactively [Music] yes there's a really cool talk called weld which is you know research project at Stanford and what I think is really cool about it is I love this idea of creating high level abstractions that allow the system to kind of automatically do optimization under the covers so with weld what they're doing is they have this intermediate representation where a system like spark could produce this rather than producing Java code or assembly code but the cool thing about this intermediate representation is it kind of reasons about parallelism is a first-class concept so you can then take this and map it on to multi-core systems GPUs whatever you know actual underlying execution substrate you have [Music] yeah so there's some really cool stuff coming for the Delta Lake project in particular we've been working on this thing called Delta pipelines we're hoping to open-source in the next couple of months the kind of core idea here is once people start using Delta and streaming they almost immediately go from having one table to ten tables to 100 tables to a thousand tables that starts to become a management nightmare with Delta pipelines we allow you to express your entire ETL job as one kind of holistic data flow graph and give you the tools to build test and deploy that graph yeah well it's hard to beat Auckland as a location you know I'm pretty he stay for life so this is it's pretty awesome to have a kind of such a cool tech conference like right here in our backyard [Music]