Keynote IV: New Developments in Spark

Spark has a simple and high-level API, but one thing this enables is making big changes inside the engine without affecting user code. Over the past year, our team has been working on some of the biggest changes to Spark since its creation, improving both its performance and usability and debuggability. I plan to cover some of the latest updates to Spark, including Project Tungsten for off-heap memory management, I/O layer improvements, and new monitoring tools. I'll also discuss new APIs that allow even richer optimization.