Scale By The Bay 2018: Cody Allen, Shapeless Party Tricks in the Enterprise
Recording: Scale By The Bay 2018: Cody Allen, Shapeless Party Tricks in the Enterprise
[Applause] thank you so yeah this is shaping this party tricks in the enterprise all these slides are on github this is kind of a long link but if you go to github.com slash c dubs i have the repository penned right now so feel free to look at that so yeah today I'm going to give kind of a brief introduction to shapeless but not really because I don't have time so mostly I'm going to dive into a couple examples of a couple methods I've written at work over the past few years these are things that I've written that like fortune 500 companies and like shapeless can be used in the real world it is used and I want people to know that this isn't going to be an in-depth X not by any means an in-depth explanation of shapeless or an introduction to it there have been some great talks and blog posts and even a book about shapeless and so there's a resource slide and I encourage you to check that out if you want some more information about that the goal of this talk will be if you haven't used shapeless to kind of pique your curiosity if you have maybe these are a couple waves I'm using shapeless that you hadn't thought about using so yeah I'm Cody on I've been doing scale development for quite a while now I work with cats I'm on a team it does machine learning and natural language processing at Salesforce and I think it's worth calling out that when I first looked at shapeless I couldn't really wrap my head around it it's fairly abstract and I looked at it and I was like I'm pretty sure it's doing some cool things I just don't really know what they are and it was until like a year later when I had a good concrete use case for it that it started to click for me so if you've looked at shapeless or like watch to talk about it and couldn't wrap your head around it don't worry that's like a totally normal thing yeah the shapeless readme says that it's a type class independent type based generic programming library for Scala well that's kind of abstract and so yeah like I said I'm not gonna really have time to go into some of the details like how its implemented but I do want to show a few examples of using it and I also want to point out that a lot of library the Scala ecosystem are using shapeless again these are libraries that Fortune 500 companies are using like shapelessness is not an academic experiment I think some people think of it that way maybe it started out that way but I think at this point it's a pretty key component of the Scala ecosystem and these libraries tend to use shapeless for like type class derivation and it's really good for that but today I'm gonna focus on a couple kind of one-off methods where I think you can do some cool things with shapeless that don't quite maybe fit that pattern as much so first is gonna pulls from my current team at work where like a test that we would have is seeing an email and maybe it looks like this what's your availability on Wednesday afternoon and trying to determine is this person trying to schedule a meeting and so in order to develop a model that does that you need some training data so we might have some judges read some anonymized versions of emails and label them like hey is this email talking about scheduling and ideally they would either all say yes or I'll say no and practice they don't always agree sometimes it's kind of ambiguous whether someone's scheduling a meeting or not so one statistic that can be helpful for us to look at when we're kind of trying to assess the results of these labeling tests is what's the pairwise agreement between labelers you know if we see that hey these two labels are agreeing ninety-five percent of the time but each of them only agrees with this other person 50 percent of the time well then like not everyone has the same understanding of what it means to like schedule a meeting with someone or ask about availability so we should just kind of you know talk that through and write up some guidelines figure that out so the first step to coming up with pair wise agreement rates is to just get you know collection of all the unique pairs of labelers and so this is something that one of the members of my team I think at a pull request that looks something like this where you know you take this list of elements and you just kind of break it into a list of tuples of all the unique pairs of elements so if our label errors are a b c and d then you know we'd get back a and b is one bear and sees one pair ecstatic so pretty straightforward I think this implementation is actually based on this Stack Overflow post when one of the other members on my team said oh hey this is just combinations from the standard library and so yeah sure enough if you look at combinations if I call it with the number two then I'll get back all my pears a and B a and C etc etc and it's a little more flexible and that you can call it with you know an integer three and get back the pairs of unique well I guess enough pairs triplets of unique combinations so I thought oh yeah this is cool this is more flexible but then the original author pointed out oh well this is actually a little awkward because this returns me like a list of lists where as in mine I'm really I really only care about pairwise agreement so I was getting back a list of tuples the list of lists is more awkward to work with because I know there should be exactly two elements in that list but like how do I prove that and you know that's not really the type safety that we'd won as scholar developers so yeah the original person who had suggested combinations like oh yeah good point you know as you were but I kinda was thinking hmmm can we do better than this can we have like the best of both worlds where we can say whether we want you know groups of two or groups of three or whatever but not have to give up that type safety and that might sound like a little far-fetched right because we don't know whether this person is going to say 2 or 3 or 20 so how are we gonna do better than returning a list or Sikh or something like that well it's possible maybe I mean it's conceivable because it turns out that the compiler actually for literal values has kind of special handling gives those special types and you can see this if you create final vows so if I create a final vow and assign it to some runtime integer value like available processors and sure I just get back an int but if I assign it to a literal like 3 I get back an int but it this refined int type where it's it's furthermore at the end three and the compiler does actually keep track of that information and shapeless actually provides a way for you to take advantage of that and use that to do some cool things so I crafted up this little concoction it's this will look a little messy to you especially if you haven't you shapeless maybe even if you have you shapeless but I want to point out that oops this first part here is just like type parameters so there's not like anything particularly interesting right in here and they're actually a little more complicated they need to be that's been changed and will be in the next shapeless pull request but basically it's taking a natural number which is basically like an ant but we've keep this kind of literal value of which int it was and your list of items then it's just taking some implicit arguments so you know just some type quest instances nothing too fancy there really other than some may be awkward types and then it's just calling through to the standard libraries combinations and kind of running that through a transformation this good or else is okay that's a little clunky but this is just if we tell the student library hey I want groups of three and for some reason it passes but us back a group of five we kind of have to handle that and I don't really want to return an option because I know the stain of library is not actually gonna do that so I just you know yellow throw an error so okay there's like maybe some kind of type noise in here but if you look at it like this is really just a one-liner I think when you kind of ignore the crust and and if you call it you can see we call combos with two we get back you know the list of all the unique pairs like we would expect if you call it with three then you get back you know the triplets but the cool thing is check out the return types so when we called in two we got a list of a tuple two of strings and since these are representing ssin between compile-time and run-time it's a little blurred well I can assure you that like these static compile time type of this is a list of tuple two and when you pass in three it's a list of tuple 3 and we can do this because shapeless is keeping track of this literal information I might be cool some time can I come back so yeah and we pass before we go back at to before and so to me I think this is kind of cool because we had you know a 1-2-3 liner depending on how generous you are that gave us gave us the both of this worlds we can use different integers but we can still retain all that type safety that we'd want so I use case that came up a previous job we had a client and a server that spoke via WebSocket protocol and we had an API on the server where you can say ok now I just want to pay synchronously await a message from the client and so for the purpose of this demonstration I'm just gonna say awaiting a message is where you just like sleep for a millisecond and then return some random message in this case I'll just use the current time Millie's so this is just some IO stringing where the string is the message and you know we can run it but then you know some time passed you got into states where okay well I want to wait for two messages from the client and I need to do something once I have both of those results ok well that's easy enough with us we can do a for comprehension are there a couple ways to do this but if we're working with IO right we can just do this and then return the tuple of the two messages and we get back our IO of our tuple two and you can see we can await two messages sure enough some time passed and turns out oh well when we get is one state we actually know eight three messages from the client because then we're gonna do some stuff and so okay you can throw another line in the for comprehension or if using cats like there's either even some other syntax that's pretty nice like this is still pretty concise especially maybe the second example but this is the time that my coworker at the time Randy who's actually sitting over here hi Randy he came to me and he said you know there's there's clearly a pattern and he had the intuition that shapeless might be able to do this nicely and I thought that was a really great idea but before I dive into the shapeless I also kind of recognized this looks like replicate a we call it in cats I think it's replicating in in Haskell where basically you take some applicative functor and you repeat its effect n times so here are like we can take our await message we can replicate it twice and you know we get back these two messages we can replicate it three times we get back these three messages but it's kind of like in combos before where we know we're expecting exactly two or exactly three but we get back this like list right so yeah I thought I think we you know we agreed we can do better and we wrote up a more general version of this that preserve the type safety so this became replicate H which is actually losing kittens now if we have time I'm going to come back to the implementation but I want to show some examples of using it so if we take our same await message and you know replicate this twice with replicate H then we still get back those two messages but you can see the result type is an i/o a pool two stream and similarly with three a tuple three of string it's it's a similar pattern to before right the one difference here is that the the combos method that we wrote was just calling through two combinations in the standard library and kind of doing some stuff with the result it was pretty specific to that use case this replicate H is actually really general this will work with any applicant of func Terr and it turns out there tons of applicative functors so what we did it with i/o that's cool what else can we throw at it let's throw a list at it so we have the list of you know the elements 1 & 2 we replicate it twice what does that do well it gives us back the pairs of two you know two element pairs that turns all of those this is different it's looks extremely similar to combinations right but the difference is it'll pair items with themselves so we get one paired with one what you didn't make sense for us when we were talking about labelers because every label is going to have 100% agreement rate with themselves that's not very interesting but this is just in you know when you turn it through the crank this is what replicate it does on a list and yes similarly if you put give it three you just get back the The Triplets of elements right okay what other look at it functors can we throw out this let's try out state so this is this this state here will have some integer state value we're going to take it we're gonna return what that what that meant was and we're gonna say our our new state is now just whatever was plus one and so here I'm not doing anything with replicate a replicate H or anything yet but you can just say this is just basic state operations if I run it with zero I'll get back okay well your new state is 1 and the value that I return to zero let's throw a replicate H at it so I have a few different states here where all I'm doing is changing this from 2 to 3 to 4 here right and there's a corresponding change in the result type from a tuple two to a tuple 3/8 of a tuple four and when we run them you can see that the like when we run the one that replicates the effect twice then sure our output state ends up being too because we've incremented twice along the way it shows you here hearing the values that I saw while I was doing this the first time I returned 0 the second time it returned one and it just collects these into our tuple two in this case our tuple 3 in this case and our tuple for in this case and as you would expect you know if i replicate the effect of this four times my final state ends up being four let's just keep their own some more stuff at it so robson over here has this auto library a parsing library and so it has letter which is a parser for a single character and he has a Combinator in there called many n where you can say hey you know I want to repeat the sparser three times so yeah if you throw like a PC at it you can see it does pull back a b c similar to the pattern we saw before we're getting back a list even though we know it's we expect exactly three of these things he has some other syntax where you can use this little squiggly operator and so you can do you know letter squiggly letter squiggly letter and that works and it has a nice type safety it returns back this nested tuple which depending on your use case might be fine or might be a little bit awkward but I think it's cool to see the replicate H also just works pretty nicely here let's take this you know parser for a character replicated three times now we have a parser that when we run on our string we get back to pull three of the three characters that it parsed let's see I think I have enough time to take a brief glance at replicate H implementation let's see my yeah it's not there I need to put it on my other screen okay that's really small isn't it okay so this is there's a replicate age this I mean the whole file is 38 lines long which isn't trivial but it's actually not terrible for something that's this general I think and does this many different types of things and if you use shapeless before you start to kind of filter out a lot of this mentally as a boilerplate right so like oh here's my actual type class this is some ox pattern stuff that I ignore that here are the two type class instances that I actually need to be able to do this and then really the only logic is like right here I just call through to the two type classes one after another and it replicates this thing in the way that I want so I don't expect anybody here who doesn't already know tables works too to see this but what I want to point out is just that you know it's it's a lot of white space a lot of types but like the actual implementation this is fairly trivial once you ignore that stuff I think it's like this little call one thing and then call the other thing and so like if you think this is interesting check it out more you know I think learn more about shapeless this is just kind of whet your appetite okay so can I get you back to the other screen that's the question all right I computers are hard so I really just have like takeaway lessons on another slide so yeah maybe I can the problem is my terminal got into full-screen mode oh yeah okay all right computers so alright so if my takeaway lessons really are just one is that if you're like oh shapeless is cool I want to use it but you don't have a concrete use case I think it might actually be kind of hard to wrap your head around it was for me if you can do that that's great but I would say viewing it through some concrete lens is kind of the easiest way to learn it so when when is it that you would want to use it so if you are if you know something at compile time and you're telling the compiler like I know this but it doesn't know that then that's often a time when like you can use shapeless to kind of get some of that compile time information known by the compiler and yeah I like that there might be these implicit sand these type parameters that get kind of noisy but I think the actual implementations are often like very long little logic just kind of a lot of type II instance a boilerplate yeah so thanks to all these people I'm running out of time so I think we have a little time for questions yes so this can be an issue compile times can go go up significantly in practice it hasn't been a problem for me I have seen it be a problem for some people I'm eyeing someone in the front row here if you like try to completely use co-products instead of like kind of the standard way to do some types in your codebase like everywhere then okay like it can it can add up to some really big compile times if you use it for this sort of thing here and there and my experience it's been alright like we use do be all over the place which is using this sort of stuff for some derivation and like cases [Music] one of these methods like ten combinations method that you chose that showed that expects a literal mean if I pass it something that is a little bit too I get doesn't work yeah I think it with this implementation it would fail to compile it would probably I don't know whether it's desirable I would imagine it's probably possible to write one that kind of gracefully falls back to giving back a list or something but I don't know if that's really good better or worse behavior yeah let me think about that I I'm not quite sure like I I would have to think about what the like non-type level version of that looks like like is that like a fold in maybe or something and I would imagine that if you can do it in kind of the non type level away non type local programming way then you could do it with shapeless probably but I don't know that's something I have to look at and try out [Music] so sorry I've been forgetting to repeat the questions but yeah the question is basically when it knew that this int literal was specifically like that literal 3 is that because of the JVM like int caching no this is not related that this is all in the Scala compiler you know and it parses your file and it sees you know builds up the ast and it sees like oh here they provided a literal and value and it stores some extra information about what that was [Music] yep okay so the question was basically is this specific to like in literals or is it kind of any static compile-time expression like I've had three plus four so there are probably other people in the room who could answer this better than I can but I don't think like three plus four is something that has like that it knows that static compile time is seven but it does know if just for any like any primitive and I think also strings and symbols but I think those are going away so yeah those and the string literals is something that shapeless actually uses quite a bit to represent like case classes in a more generic form where you have kind of a literal symbol that's the field name [Music] [Applause]