Scale By The Bay 2021 : Eduardo Gonzalez, Introducing Spark Cyclone Accelerating Spark with...

ai.bythebay.io Nov 2025, Oakland, full-stack AI conference Title: Introducing Spark Cyclone: Accelerating Spark with the hidden supercomputer device plugin in Hadoop The Spark Cyclone project is an open-source plug-in for Apache Spark that accelerates Spark SQL with the NEC SX-Aurora TSUBASA accelerator which is supported in Hadoop 3.3. Dubbed the “Vector Engine” the card has lots of onboard RAM (48GB), lots of memory bandwidth (1.5 TB/s), and lots of computing power (6 TFLOPs). This talk covers why the Vector Engine is uniquely suited for analytics (compared to the alternatives), how we execute Catalyst queries on the Vector Engine, and how the performance compares to Spark Rapids running on a V100.