Spark Datasets: why they aren't great an...

The Dataset API enabled Spark to implement advanced optimizations that were not feasible with RDD, but it also introduced a less intuitive and type-safe user-facing API. This talk will explore the reasons for the Dataframe/Dataset compromises and present a potential solution that would enable Spark to provide a fully type-safe and more intuitive Scala API through quotations without losing the ability to apply advanced optimizations.