SF Scala: Nadav Samet, ScalaPB: Working with Protocol Buffers in Scala
Recording: SF Scala: Nadav Samet, ScalaPB: Working with Protocol Buffers in Scala
oh hi everyone my name is nadav samet i'm a cto and co-founder of two accord before starting to accord i was at google i was a software engineer there working on a few projects like gmail google apps google flights google hotels and i was writing a lot of protocol buffers over there when i left google to start my own company one of the things i discovered is that we are on our own we have a very limited time to get a minimum product to market and we want to leverage as much as two as many tools that we can find that we're comfortable and we already know and protocol buffers was one of them for me we started with python and protocol buffers have an official support for python by google and as we move to scala we shifted to the java implementation that google provides for protocol buffers and the cool thing about it is that the java and the python wire forward all the wire formats for the profiles there's a standard they're all the same so the scala server and the and the python server could just talk on day one so i'm here today to talk to you about a protocol buffers in general what is scala pb and why we wrote it when you should use protocol buffers and when you should not give you a little bit of a test of the api and then talk about few specific things that i learned while i was developing it so you know one of my goals here today is that even if you don't end up using protocol buffers called pb which i think many of you by statistics will not use scarapb by the end of this year you'll still learn something from this talk some of the things that i hit while developed it and some things i learned about functional programming in scala that would just be useful for you as you go along so let's get started by the way before we start is there anybody here that works right now for google great so i can tell you everything i want so um basically there are two types of jobs at google for software engineers there are people who take protocol buffers from one format and write it to protocol buffer in a different format and there are people who manage the people who write protocol buffers in one format and write to another format so uh i hope your jobs are more interesting but basically a lot of times this is like a kind of truth right in a lot of corporate jobs you find yourself working with data formats and moving data from one data set from one source to repo transform it a little bit report it or write it to another place so let's you know find a way to make it nice and unpleasant to do so protocol but first let's start with the definition and we go with an example in a bit so protocol buffers is a language to describe data structures on top of the language there's an implementation that you that defines the encoding how do i take those data structures and serialize them as bytes or how do i take the bytes back and get the original data structures and protocol buffers comes with a compiler that generates access class for java c plus plus and python go ruby javascript other languages and with scalpy bit gets color support so this is the example this is what the protocol buffer looks like i hope this is large enough um this is an example of a person object and it has three fields the first two are type string the first name and the last name and then there's an age field at the at the bottom and every field comes with a tag number at the left at the right side it's a one two and three and when the message gets serialized on the wire it says key number one that's the value key number two the other value the last name key number three there's the integer that represents the age of that person and everything in this specific message is optional so the parsers they expect that you know some data will be missing so you write this file then you write this is the protocol buffer compiler it's called protoc and you give it a protocol buffer and you tell it create a java class a java source code out of it and then you get a giant file and basically you have a access accessor for this protocol buffer in java we call java from scala in this code sample so we everything the java implementation is immutable and you have a builders to create those immutable objects so you have a person you call new builder you set the first name the last name the edge you build and then you can say give me a byte array and to get it serialized we convert to vector because color is a nice way to put vectors out of the other if it was java it would just be like a like a the memory location so this will be the byte presentation is a vector of bytes and you can do it exactly the opposite of that if you start from an array of bytes you can do parts from and get the original protocol buffer back and protocol buffers in the java implementation come with a nice two-string method that would give you a nice it's very nice for debugging when when you're able to print the messages so another feature of protocol buffers is that you can nest nest the messages inside one another this is very useful to get you know to build more complex data structures so here we have an address of a person which is just a street in the city we have a bank account which is a routing number an account number and then we add to this person at the bottom um the address as an optional field and a repeated bank account list so this would be a a collection of bank accounts that this person has yeah why is there um routing number why is everything optional because bank account without one of these numbers make some sense right so we leave it so the idea here is that there might be some evolution happening to your protocol buffer over time and maybe you'll decide that you want to represent it in a different way we live we don't want the parcel to crash we want the application to take the data and figure out whether it should continue with that optional thing or not so we'll touch evolution in a i think in a bit um so um you shouldn't use protocol buffers all the time but these are some good reasons to you know to kind of figure out when you should use them so and why not json which is a great format and very readable and very common so it's you get an efficient binary format so we don't we when we serialize protocol buffer messages we don't have to repeat the keys they are just numbers so it's shorter version shorter way to represent them and the numbers themselves are in a binary format it's a very very efficient um you get classes that to get to to access the data so you have this person object which you can access the fields of it programmatically rather than you having a dictionary of strings of some sort um but the bigger benefits are actually what's coming next so let's say you want to set up two servers and you want them to communicate in a binary and type safe way you can use protocol buffers one of them can serialize the message messages other one can the other one can pass them and you set up type safe way to to have the process interact and the nice thing about it is that as your protocol measures evolve if you use optional fields you can have two servers that compiled against different versions of the protocol buffer running at the same time and uh without you know having the program crash which is something we're gonna touch and another beautiful thing that you get from these protocol buffer files is that you have a schema that you can share in your company between teams and you can tell them this is this is the data structure we have when you get the person object this is the field it has these are the types you can add comments between between the lines and say this is what this thing means to me um you can use it as a persistent storage format you put a protocol buffer in a file you can read it later on it's exactly like communication but when uh as but you know more long term so as a data there's more likelihood that the data will evolve by the time you read it and this is a i think something that a lot of people don't mention when they talk about protocol buffers is why they're so great for evolving schemas and this is a little bit of a story about us when we started this company we had no idea where we're going we knew that we were going to do about account recovery and and and we'll have an account object and a person object but we didn't know about the relationship so as we started we we the account object was a central thing and we thought that every account is a person but as we go we started to say to see that we actually have multiple a per we have persons with multiple accounts and we have to change our data model a little bit so here's an example of what you can do right so you have a person this is the before we started evolving it and then this is what you it can be after you know as you grow so you decide that you hate first name and last name so you go with given name and surname and and then you don't like the age field because it's a bad idea to store ages you want to store a year so because time goes by and the eight changes and and you do something like that and what happens is that if you serialize a message with a parcel that's written in this format that one can can read it so it will access because it has the same one is the same tag this one would when it tries to access the first name it will get the given name that was set in that message so you can rename fields at any time you can drop field because it was assumed that this is optional here and you can just add things and this will ignore them if you try to read them so let's talk about things you can do it's add optional fields remove optional fields rename fields you can convert between compatible types so if you have an in 32 you can set it to in 64 and if you have a parser that expects a in 32 and you call it e6 in 64 you might get the data your data truncated but that's it i mean if it's not a concern then you can always do it you can even make it a boolean and get just a zero one um you convert optional fields to repeat it so if something is optional you make it a vector it can be a vector of zero one when you pass that message you can also get the other way around if you said something's repeated and you want to downgrade to an optional you'll get the last thing if it's not empty sometimes you figure out like you know we have we have a payment plan and we want to have a to add a payment method of payment to a payment plan then third actually we want to have more than one so it's very easy to just say oh this field is not an optional it's actually repeated but there are things you cannot do you can't change the field type in an incompatible way so you can change the integer to a string it will just mess up the format you cannot remove a required field so a required field is something i didn't show but basically if you said that the field is required when you try to parse it if it's not there the password will crash it will just throw an exception and you cannot add the required field because maybe somewhere in production there's gonna be a you're gonna there's gonna be something that still makes the old types of messages now your new pastor expects the required messages and it will crash so in general the the conclusion that many people came to is do not just don't use required fields if you care about the existence of a field just checking your application level don't rely on the protocol buffer implementation to do it for you and the required fields are going away anyway so in portal 3 they're just not going to be supported so um i want to show a little bit of what the java photo buffs looks like and this would give us kind of a feeling why we want to do it in scala so you start with the builder and you set the first name and last name and then you get the address builder because i want to set the address of the person and i call set street and set city and build and i expect to get a person right no what happens is that i have an address builder here and when i kill when i call dot build it builds the address of that person so the indentation that i put it doesn't help it's not python right so what i get is i i i get a compilation error and i get it says you have an address because you built an address used you could build on an address builder you wanted to get a person so the right way to do it is this you basically set a person builder you have to put the builder aside then you get an address builder you populate it then you call build on on the other one and you can just say it adds like intermediate variables it's not as nice as it could be and another thing that bothered us about protocol buffers in java when you do scala is that java back at the time didn't have support for optional fields so the protocol buffer implementation in java says uh it has what it said the ghetto setters and hazards so an ezer is a method that tells you do is that field is actually set in the protocol buffer so we had a lot of code that looks like this if it's actually reversed if it has it then it's some of that bank account but if it doesn't have it you have none the logic here is flipped and this is actually one of the reasons we changed it it's very easy to flip you know booleans and get a reverse logic so now that we are we know what protocol buffers are let's talk a little bit about scala pb and how it fixes some of the problems we saw and create a nice api for for a accessing protocol buffers in scala so scala pb is a plugin for protoc that generates case classes from your protocol buffer definitions and it solves the nested fields up the update of nested fields with the with the technical lenses that we talked about um it adds converters to and from java so if you're already stuck with the java protocol buffers and you want to switch to scala it's going to be very easy to live with two of them you can just compile both of them into your projects and scala pb will create converters to and from those classes so you can just gradually pull more stuff to the scale implementation and it's written as a protoc plugin which is an interesting property of the system i let protoc parse the protocol files i didn't write a parser for the synth of the for the portal language so the protocol does all the parsing and the scala pb takes the the tree structure and creates the case classes so this would makes it much much easier for the implementation to support more features of protocol buffer as the language evolves and this is why we can ship proto3 very quickly probably next week so this is what it looks like this is the message person that we had before and this is the case class you get from it so everything that was option optional here becomes an option there with the default of none and you get a two byte array method and at the companion object you can parse an array of byte and get the person back this is essentially it so when you start to when to build those objects this would be the default way to generate the case classes so you have the person you have to assign things you don't what you don't assign gets the default the none and here's how i build the address right i have the address and i have a sum address and this is better than the builders but there are a lot of sums it's a little bit dirty another issue that you have with case classes in general when you work with immutable data is that nested up updates are really hard and um for example if i want to change the street of that person i need to make a copy of it and i need to get the address and i want to and i ca it's an optional so i need to get and i copy the address itself just to change the street and this is all by using lenses you basically get this syntax which is much much shorter it just looks like a update with an assignment so i'll talk about like how this is done yes so it generates and it doesn't compile yeah there are ways around it i never had like a we never had to face it with the system but there are ways to get some i mean there are libraries that you can use that would make it compile uh i i just never touched it but this is a solvable it's a technicality it's solvable and so let's talk about nested updates um so and this is the classic example of how the product lens lenses solve so let's say you have this structure you have a cluster case class of a that contains a b the b contains a c the c contains the d and the d finally contains an integer e and this is how you initialize it and this is nice so far but what's what how do you now change this a this e that's deep inside the structure to 17. this is how you do it right with immutable updates you have to copy a to copy b inside it to copy c inside it and then finally change d to f 17 inside it with mutable updates this is just this is just it right so it's i mean nobody would think that this is better than this right at the same time like what can we do to get something as close as possible to this so lenses is the the solution to it and i'm gonna give like a quick introduction to it with without scary words without category theory without or without all the things try to make it something we can all grasp at the 7pm so um we have lance and the lance is away he's a getter and a setter from for a field inside a container so if i have a container some person and i want to get the address out of that person i can have a i i'll have a get method that will get the person and give back the address the settle will take the address and the person and it will return a new person with the updated address and to make things nice to to write we just have an alias column equals is just set it's this is the assignment so this is a concrete um implementation of the of the of a lens that access the address of a person it's a little bit truncated because i increased the font at the last minute but bear with me um we have a get that just takes a person and gives back the address we have a set that takes an address and a person and copies you get a copy of that person with a different address shipped through that parameter that's a lens so this is how we use it right you have a person address lens and you want to set a new address this is how you call it it's exactly it's just applying this function and because we have the column equals shortcut we can just write this instead this is exactly the same thing now the beauty of it is that lenses mathematical structures they compose you can just compose them with one another so if you have a lens that can change the address inside the person and you have another lens that can change the street inside an address you can combine them you can compose them and you get a personal street lens so you can access you can get and set the street directly from person this is just function composition no magic and this would allow you to do a to update the street directly through a person with with keep skipping the address part so you have a personal street lens and you just assign a new street to the person and you get the updated person so scala pb would just generate those lenses for all the fields for all the objects with enough implicits and magic and smoke you can get um i'm not sure what was the difference between those two lines ah so this you you can get something that you can access the field in a nicer name not with the with the big companion object is there anything special for you that's uh the address of the whole office so actually this is a backward compatibility yeah yes um so with enough implicits and traits we can get we get access to this syntax which i find really really nice you get a person and you can start from an empty one because everything is optional so this is initializing the person case class with nothing and now you start updating it by just sending those just calling those setters and it it's written in a way that it will update this object it's like a fold left they're just running these functions one after another so at the order of those things metal and you can just access and assign something for the first name if you have the last name as an option in your system and you just want to assign it if you have it you can use an optional last name you can set this thing directly like a nested field director this is what we're trying to actually solve skip the builders stage so this is letting you go directly to a nested field of at any depth level and assign a value to it you can do mutations so a lot of times you want to take something just change it in some way call a function on it so modify is a is a taking the value of this and assigning it again so it's taking a value from a string to itself if you have repeated fields you can just combine you and you have a sequence of accounts you can just um append to your list and this is how we can do like you know a lot of time when you have account numbers you want to mask them so this is how you do it in um over here without copies and without going into the the the nested things that we saw earlier so you get a very concise syntax for mutating a nested structure and this has been you know we just when we moved to this we delete hundreds of lines of code that was just like build and get builder and and all that stuff so um i want to talk a little bit about testing as well which i also learned a lot of things while while i was testing this library so think about you know one of the scale thing for somebody writes a label like this is to open source it and then people send you hate mail because their data is lost so i wanted to avoid that part of my life and ship it version one with a lot of testing so think about the testing plan for this you have 15 primitive types you have enums you have repeated required and optional fields packed repeated fields which is a a way to put the bind like a repeated field of primitive types in a more complex format nested measures messages you can have default values for fields you can import other product files from different like if you have a lot of qualifiers you can import them there's something called one offs which is a like a c sc union type the maps that was introduced in photo 3 and you need to be able in porto 3 to import proto2 files so you have two versions of the language that can import each other and you need to make sure that your code tests all of these scenarios because things happen when you just combine those features they can interact so there's a cool tool called scala check that if you haven't tried it ever then you should basically what it does it can generate random data for you and you set rules or checks that would pass only if sudden condition old so for me the conditions were do we get the same behavior from java and scala so i wrote some code which turned out to be very complicated to generate random protocol buffers it's complicated not because it's complicated to generate random data formats but because a lot of rules of how you compose it so i want messages to refer maybe to themselves but you can't do forward references so you have to keep a list of what you already defined and you can't call things from the future um and a certain name so you just you know you cannot use them turns out that the java compiler crashes when you use certain names it just i mean what i mean is it generates invalid code when you certain names um so this is basically a large tree of files and with a lot of imports and it creates a basically a a very complicated test case then what you what we do is to generate random data to fill this protocol buffering and and get it get it serialized by java gets realized by scala are we getting the same thing um this is this was testing such by running it on proto3 i was able to find bugs in google's implementation of all the three i reported on github to probably take care of it sometime soon but it it was basically about the case where you have a map and the value type was exactly the same type that contains that entire map you get a null pointer exception so this is a very very powerful way when you have something with a lot of options with a lot of complexity to just generate a lot of test cases for it and just make sure that you're covering everything that's possible this is what i had um if there are there any questions yeah i have you liberate on the use cases for this uh more details sure so uh let's say that you are you have two teams in your company and you want to set up some client server relationship with one another and you want to agree on what you're sending so coming up with a structure with a with a document that says i'm going to send you messages and these are the fields that they're going to have this is what they mean and never another way for them to read it and access it all the time um is something that proved for our company very very valuable because it serves just as a documentation of the of the data structures so it's very good for internal uh communication between teams in a company i wouldn't necessarily use it for you know if i did a public api i'd probably use json or something more you know that people can easily always familiar and can easily access it it's for me it's more like a tool for persisting data or for cross-process cross-platform communication and uh since probably if the application is monolithic then it probably doesn't make sense to use them right even if the application monolithic there might be good user case for example if you persist and you want to store the data and you want to know you want to enforce the type safety of the data that you that when you do it it might make sense drift yes photobuff why did you choose so i so all of them are very good tools and they are comparable you know they have some of them have features that the other one doesn't have i chose protocol buffers because i was very familiar with it with the years of you know using it at google um so i just want to look for something that would be you know make me productive on day one what was important to me and also all these tools have is cross language support part of our company the data scientist use python and we wanted to do something that would work with that have bindings to as many languages as possible so we are not get locked into anything uh but all these tools are valid and everything that i told you today applies to them as well so a lot all the reasons to use them and not use them apply to them they're just alternatives yes very random question so we have as i think most people have used roadbus for a long long time gotten in the classical tons of required fields here and there oh and you mentioned they're going away in three do you have any idea how painful that transition is so the good news is that they're not so they're going away in potter 3 but proto porto 2 is going to stay for a really long time and there's just there there wouldn't be a strong incentive to to transition just like if you write a new profile write it in proto3 there's no there's no incentive or big win to move to convert your entire system to port 3. how difficult the transition is going to be it's a good question so i think when you compile it for the first time some things would not compile the comp if you use java or scala right the compiler will tell you actually only if you scala because in java the distinction is a is a is not as you know you just you have the hazards that if you don't call them the compiler will not be able to tell you but in scala we will basically move from option of string to a string so things will just not compile if you change from a required to option the compiler will save you which is part of the reason why we all use scala right the compiler saves us yes whenever we get in trouble or we before we get in trouble yes so there's the protocol format have a version number like somewhere in the first couple of lights that that was used version two version no it doesn't have that okay yeah so so how does i mean what's the difference that um the wire format is very compatible so what they did in porto 3 is that they added new features to the language for example maps and there's any type so so if you start to write to use a map in proto3 at proto2 it will be a list of messages that each one of them is a key value pair so proto2 parsers will be just able to read it exactly the same way the differences in the interpretation of the word right do you see people over time putting version numbers in their messages there's something called self-describing messages which actually contain the the actual definition of the message in the message itself so a parser could just take it i have no experience with that but it's useful for some people like containing entire definition or like version to actually have access to that yes but i imagine that the usual use cases as you said internally right with one thing that already has a system that's everything then there's a new team that their expertise is in scala another one is communicate can you use so the java group already has the data classes can you use protobuf to generate from the existing java classes and compile the protobuf classes so that they generate the scholarships we should start with protocol and generate both so the existing to repeat the question there are existing java classes and you want to add protocol buffers into the mix and you and you want to make sure that so they'll probably need to you'll need to write some custom layer that would convert between the old plain java classes to the protocol buffers to the case classes or to the java portable buffers depends which one you use sometimes what we find is that you know you get those automated classes from from the protocol buffer compiler but you want at some point in your application to move to your objects you don't you know these objects are very shallow sometimes you want to add some nice methods on them with sky it's really nice because you can add like implicit conversion to them and just throw more methods if you need but a lot of time what happens is that you take those case classes that you get from scala pb or the class you get from the java compiler and just write your own business logic with a different set of classes you just have converters between the two my understanding photograph is there are definitions and that can generate classes in many languages right right so now you can have multi-language systems but the normal use case is that somebody already has a system and has the job implementations and then there is uh scala implementations missing from another system so you can go from protobuf to a cloud alignments can you go from alignment but that's not something that that comes with a library because that would be rewriting in both senses right you have to write the definitions in program and then replace in both systems the corresponding classes yeah if you have every investment in existing case classes there are probably other tools that can uh that you can generate some civilization the issue with with those tools that they probably expect some version of it on the other language and then you you have less choices about the languages that you can have on the other side yes so as you mentioned everything's optional uh makes all the fields options right and that can go in the way a lot when you're making when you're writing the application code is there like a way to make fields required and like avoid this option craft yeah so what happens is that the co i didn't show it but the if you know that your field is there you can you you can call you have the message and instead of calling dot person which give you an option person you can call get person and get the default object if it doesn't exist so an empty object or the actual object if if it's if it's there so this is better than person.get which would throw an exception if it's if the option is none so uh so this is this already so you don't have to go you can go through these options very easily options right yeah and there is no way to let's say uh enforce uh final types and then maybe have run time exceptions so portal 3 is for you so in photo 3 they through the all the required fields and there's no meaning of an optional field because if you so all the fields are optional so there's and they what they did is to the they get got rid of the notion of whether the field is set or not so there's no way to check if the field is set or not in java or in the wire format so if it's not there it's assumed to have the default value so basically you get a zero for all the integers and empty strings for all the strings and then you if you use proto3 you'll see much less options the only time you see things wrapped in options is for nested messages any more questions yes so one difference with everyone will notice this so ever has separate schemas and so from what i understand if you if you want to support different versions of portable buckets you have to have the the compiled versions of all of these versions compiled on the glass part right so um so so in reality right let's say you evolve your schema what happens do you actually so if you go to my micro right you call it something right and you have to rename it how do you like the different versions so how do you access multiple versions of the same so you know of let's say a card right account and you add some fields to it and then you text all the new rights you have the original product defined as a name then you compile the generators of java clusters so they have a name account in them now you compile it again and have a different thing on the column it's called how you manage the existence of two different uh puzzles yeah so when you when you change something uh in a protocol buffer definition you always have to assume there might be still pieces some servers in production that speak the old format so you have to be defensive defensive of that in your application logic you say what happens if i don't have this field should i bubble up something for the next lesson should i get my either should return something on the left side or should i for exception it depends how you do your logic um but it's basically it's a something that protocol buffers say we don't have an opinion on it's the upload application to to guard itself from different versions so it will not crash it will give you nuns in in the optional fields but you have to write code say oh it's not there it must be an old thing go to that code path right what i mean is the opposing names you basically you always have the new version only compiled in your application right you don't have all the new compiled together because uh they have the same name right so you keep the name of the uh of the class right uh and so sort of like you you create optional fields and such an order so your students will fully subsume the previous thing right but you don't have two classes compiled into the same yeah this is not something that's currently possible i think with with the count implementation javascript you can't have the two versions of simultaneously in the same game that's right i was just thinking let's say you want to actually know so you can go and rename the whole thing you know and suddenly compile the classes and do something depending on what you want to do right but then you have to manually basically support uh models what i would do in that case is basically copy so if you have the person version one you can just copy the thing and call this isn't just you know create a copy of that file with a different package name or rename the actual message names and then because you you just use the same tag numbers the same numbers you know on the on every field it will just be able to to read them so basically a protocol buffer message is just a key value store from the tag numbers to the to the values so as long as you have things that compatibility will just read it right but you will have to manually uh manage package name spaces to lesions if you want to do that so it's a minor i don't know if you guys have similar experience like i understand that ever allows you by detection of the schema our like gives you more manageability for this right like you will not have to manually manage language artifact but you will have a separate schematic that's kind of what we figured out others not just the same difference yes so again on the use case seo you mentioned that you started using protocol buffers as soon as you started your own company right so you probably didn't have teams and all that and different applications so can you elaborate on what was your use case and why did you decide to use protocol yes so when we started and we have we have one of my co-founders here it is exactly the same name as me this is a very interesting coincidence it's also called nadav summit so we go internally by senior and junior and so we started with with the with the you know we were all the first lines of python and we knew very early on that the the data is going to evolve we started with a little bit of mysql and um we we set up the data structure we said oh this is going to be a lot many many too many tables so for example a person can have multiple accounts and you need to set up a many too many table and as we went along there were so many schema changes that we figured out that we should throw at least some of the fields that change rapidly into protocol buffers and this was great at capturing the changes in the system because it was so much easier to migrate schemas of protocol buffer than migrating managing schemas of of a mysql tables so we start with a little bit of a of mysql with a lot of tables and then we start to move part of these tables to protocol buffers and now everything is a very denormalized yes we have a we use some it's inspired on the friends field approach we basically use a mysql as a key value store for protocol buffers any other questions so before we continue for the next talk uh i wanted to say that we are hiring if you want to move protocol buffers from one form or to another in a different company then you you're welcome to join we are so we are using scala for all our backend staff akka play we are solving a real tough business problem and if you want to join a stellar team of soft engineers here at the bay area and in san francisco or in san francisco just join us and i'm sure you won't regret it it's we're building something really great again between applications between languages also you will you yeah i'm not sure we're gonna go on that adventure probably new protocol buff is gonna be with version three because i really like the map use case uh but yeah there's a a lot of interesting things that they we're working on that would probably be uh if if anybody anyone here is interested to hear more about it i'll be hanging out here after the meetup uh you