Scale By The Bay 2021 : Jean-Yves Stephan, Apache Spark Performance Tuning Session with Delight

ai.bythebay.io Nov 2025, Oakland, full-stack AI conference Delight is an open-source monitoring dashboard for Apache Spark displaying system metrics (Memory, CPU, ...) aligned on the same timeline as your Spark jobs and stages. It's a great complement to the Spark UI to help you troubleshoot the performance and stability of your applications. Delight works for free on top of all commercial and open-source Spark platforms. In this talk, JY will use Delight and the Spark UI to go through real-world performance tuning sessions for Apache Spark. Parallelism, shuffle, memory, instance type.. he will show you how Delight can help make your applications stable and cost-effective at scale.