Spark and Protocol Buffers - An Awesome...

Have you ever been building a project using JSON, and managing the data format became too complex? Schema changes and type safety become a big pain. I will share one of my favorite solutions we used at Google, that many Scala programmers can appreciate: Protocol buffers. Protocol buffers are Google’s cross-platform language-agnostic mechanism for parsing and serializing structured data. With protocol buffers, you define your data schema, and the compiler generates parsers and serializers for your data in many different languages. ScalaPB is a library and code generator that brings the awesome power of protocol buffers to Scala and Spark. ScalaPB takes your data schema and generates case classes, along with parsers, serializers, and even lenses for convenient field updates within deeply nested structures. In this talk, I’ll give a gentle intro to protocol buffers and ScalaPB. I will show how you can use protocol buffers and ScalaPB as your project’s data exchange format to take the pain away from schema evolution and cross-team data sharing. We will do some live coding and build a Spark application that processes millions of protobufs!