SBTB 2014, Kelsey Gilmore-Innis:: I Dream of Gen'ning: ScalaCheck Beyond the Basics
Recording: SBTB 2014, Kelsey Gilmore-Innis:: I Dream of Gen'ning: ScalaCheck Beyond the Basics
thanks guys sup how's your afternoon going feeling feeling scalared out yet or like no no you still got some time okay cool i'm kelsey uh this here this is that's my familiar cheryl uh we've got stu up here who's gonna be joining us via astral projection later on i'm here to talk and stu and cheryl and i are all here to talk about uh scholar check so i'm curious like in the room are there folks here who are using scholar check now on projects give me a hands up all right we got some folks um that's cool i'm glad y'all are here because you guys can back me up to let the other folks know what scala check is so what is scala check scala check is witchcraft um scala check is witchcraft that makes you super powerful it's a really powerful way to find bugs in your code like really powerful like burn at the stake when your crops fail powerful it's totally magic uh what is it exactly scala check is a scala port of a haskell library called quick check uh designed for property-based testing property-based testing is it's it's a little bit like the testing you're used to so this is like an example-based test which is what most of us write when we write unit tests you sort of set up some input you run some code and you verify the output um it's an imperative model you're telling the testing framework what to do and how to verify it that's useful it's important it's good to do there's a subtle difference between this model and between property-based testing that's kind of similar to the subtle difference between imperative programming and functional programming when we program in a more functional model instead of like telling the computer what to do and in what order we tend to try to tell the computer what things are we define things and that is a really subtle distinction that gives us a lot of power and there's a corresponding thing with property-based testing where instead of defining what our code should do in this situation we say that there are truths about the output of our code given a certain input and when we define it like that actually it turns out that we can do a lot of really cool things so when we define it like that we can with a little bit of magic automatically generate lots and lots and lots of inputs and let the computer automatically verify those properties we defined for all those outputs so we sort of like you magnify your code coverage in this huge way and there's all these cool things you can do with it so this is advanced scholar check it's going to be a little bit of like a flyover tour like you guys you guys see enough code at work right i think come on um i am going to show you some code but i want to kind of give you some ideas of concepts and cool tricks and exciting projects that can kind of pique your imagination what can i do with property-based testing and at the end i'm going to my my buddy stu is going to come on in from the ether and show you a little bit of like hands-on code yourself so what does scholarship do for you the library itself it generates it runs and it shrinks so it generates input this input usually is designed to hit edge cases more frequently which makes sense right like when you're hand generating input for tests you want to make sure that you've got those edge cases right but it can cover all kinds of input it takes a bunch of these inputs and it verifies properties that you define against your code on it if it finds failures and this is a pretty cool feature it tries to shrink the failure space to the smallest possible set it can figure out so if it turns out that say you know non-unicode characters break your code it's gonna try to show you just one rather than show you a whole list of things that it failed on and that's a really really powerful thing to have for you it's like you're getting like a a whole lap head start on debugging which is cool so like this is what it looks like this is a whole file this is a complete scholar check file you could run this as is um assuming that you had a project with a dependency on scholarship without any extra imports there's nothing to find outside the file basically i've written this silly little spell method and i've defined a property about it right it's pretty reasonable it's saying okay if you've given this spell the length of it is going to be the number of digits in that magic number plus the length of the words you said plus 11 for toil and trouble so when we run it scala check's going to like try a bunch of different options not just your sort of basic let's throw some edge cases that we can think about it scala check being a computer is going to be able to think of way more than i can if it fails and this is where we get kind of powerful right if it fails it's going to shrink the failure set to try to figure out why it failed so right here it was like trying some crazy stuff right it was like throwing korean at it it had some crazy small number that it was throwing at it once it got a failure it said whoa whoa backup something's wrong with this code i'm gonna try to shrink it and it's gonna try to figure out what the numbers should be and the magic number has to be positive the magic number right uh everybody knows that so we have these three principles going on here we've got a generation of inputs so we've generated a bunch of lists of strings and we've generated a bunch of hints it's running those things multiple times and then shrinking the fire space and i'm going to talk about all three of these things and show you some cool stuff that you can do with all three so generating input how do we do that well we have things called generators and you'll notice that i didn't import anything i really didn't define anything except that code i was going to run in that last file that's because with scala check you get built in a whole bunch of generators you don't have to define them yourself for basic stuff um you get your sort of uh your primitives there right uh you've got a throwable and a date those the generators for those are biased towards edge cases so it's going to try things like negative numbers it's going to try things like the empty string things that when you were if you were writing the test by hand you would try to throw at it for things below there for those that that um second and third bullet point there it can generate those by itself if the um if it has a generator for the parameter type so if you've got an option of a string it can generate it for you automatically if you've got an option of some user-defined type and it doesn't know how to generate that it's not gonna be able to generate an option of it but given that you have a generator for the parameter type it can generate an option in either tuples of all kinds even functions which is pretty crazy uh same thing with containers it's pretty easy also if you have your own custom containers to get get free ish generators for this um and then there's some helper methods like if you're writing any kind of test you're gonna end up wanting something like this like a positive number or an alphanumeric string and what that means is we can fix that test i showed you earlier by just using some stuff that's already in the box be like no i i really meant everybody i was never going to send it a negative number i meant only generate positive numbers to test this with so that's pretty cool magic sum for you of course you know we often do things with data that's not just primitive types so you tend to want to get a little bit fancy right so i'm going to talk a little bit how you can make fancier gems and it's actually surprisingly simple surprisingly readable so three of the big ones that we use are going to be choose one of and some of um and so here i'm going to walk through these the first one choose just pick something in a range and that happens a lot especially for input that has some kind of real world meaning like an age or you know a number of months or something like that you're not you actually don't need to test it over the full integer space so that lets you do it that choose is going to pick something randomly from that range if you want to buy us towards the edge cases there's another method you can use one of and some of you specify actual values here and one of we'll pick one and some of will make a list of it'll make many lists of different sizes of the things you specified so these are some little like spell testing methods you know you want to make sure that you're not casting magic with bugs in it unless you're trying to make bugs appear in which case you'd probably write a different test for that um so that's that's useful what if we want to get a little more complicated uh these last two actually this last one actually uh we've passed in actual values there but you can use it as a high order generator which is kind of crazy you can say hey i know how to generate like an ingredient so i want to generate a list if i give you that generator and it it the possibilities of what it can generate just like explode it can make so many different things which is awesome so sum of is a high order gen list of is a high order gen um one thing that's cool about this is it shrinks things as well the same way that it would with just like a list of strings so it'll say well we're probably gonna try an empty list first right that's often gonna be a thing that fails and if you use these sort of out of the box ones to put your stuff together you get the shrinks for free too that's one way to compose it um if you want to mix stuff together you have even more paths here you can compose them using map flat map and filter which is usually called such that for whatever reason so i'm sure scalable just reading that you're like those are the three magic words because when you have those three things you can do four comprehensions which is cool here's some examples basically it means that you get to you get a lot of power out of modeling your data using these built-in gens which is cool here we can make anything into a spell we can make an arbitrary number of spells if you want your witch name here we can make arbitrary which names and i've used flat map here so that i can have access to two different generators in combination if you see a flat map then a map what are you thinking you're thinking of four comprehension right that's pretty readable right you're putting together an ingredient list and you've you've got now if you like multiply this out this means the computer is capable of generating like a gillion random recipes and so if you have some kind of code that operates on this you don't have to specify what the inputs are for tests which is really useful because you know the limitations of your tests are always as far as the input and this kind of really expands that out one thing i want to point out so such that you're going to use a lot right like if you're if you're doing anything you're going to have constrained values such that can be a little bit dangerous um such that is is another word for filter so what it really does is if you say such that or choose or like anything like that if you see such that it's an int that's less than 100 right it's going to generate ins and it doesn't do anything smart about it it just makes a bunch and it'll throw away anything that doesn't meet the predicate very much like filter on a list does which is usually okay but like if you're doing something really specific you might want to get fancier with your gen because you can run out of like tests to do it'll take too long to discard all these things another thing that you can use retry until if you run into this if you're like oh it can't find any you know names that begin with k and you can do retry until it meets the predicate but i would encourage you if you're like making heavy use of choose or such that to think about especially if this is something that has any kind of user facing input think about whether you've got code constraints that that keep that from happening because you know you say that eh can only go up to 100 but then if you've got a witch using your code which is live forever that lady's 250. so she's going to break your code and you won't even know it because your tests pass which would suck so that's just like a basic sort of that's the set of builders that's kind of it that's all of them you know you guys know how to do this this is how you build lists is that you can post lots of things and scale it you can build really crazy arbitrary models of your data using this and so this is a little snippet um from eric torre bore who writes specs to um did a generator for arbitrary json which as i'm sure you can imagine how that might be useful to test how your app responds to json especially if you're parsing it and you'll notice here um i'm not going to go into too into depth into this although i do encourage you to check it out because it's pretty elegant notice that a json type it generates a json array or a json object json object calls values it's a list of other values it's actually recursive because you know java json maps can be like infinitely nested there is a stop to this that's why there's a specified depth but that's pretty useful too that's also like the thing that gets me excited about this is if you've ever written a parser for anything it's when stuff starts getting nested that my code starts breaking i don't know about you guys so being able to automatically generate lots of of weird edge casey things about that is pretty cool other things you can do with gems um sometimes the like standard distribution of things is not necessarily what you're looking for you don't want the random one you don't want just the blonde brunette and the red head so you can specify frequency with like a huge amount of um granularity you can you basically provide a histogram so you can say i want half of my values to be even and then i want the next third of them to be divisible by seven or whatever and you can you can build that out pretty um pretty specifically and in fact we ran into this in production when someone filed a p1 bug because in the app at the company i worked for they searched for beyonce and nothing came up and i don't know about you guys but that's like wake people out of bed bad uh to me so you might get you might have already figured out why we were using alpha strings to test our stuff right beyonce is properly spelled with an e with a apostrophe on it or an accent on it otherwise it would be beyonce duh um and so our our code didn't handle it well and so we would miss it in our test so what did we end up doing well it turns out we actually didn't want to test against every string and if you do scala check like uh naively and you put string in there you'll see and this makes sense it goes for like korean first almost always to the point i've had it like slow down my machine for some reason i don't know my machine can't keep up do they not have max in korea i don't know um but it'll spit out a ton of korean and then it'll go on to other sort of less common uh strings which is perfect that's what you want this library to do right because you wouldn't think of that yourself our app did not actually support korean yet you know it wasn't internationalized so we kind of were like well we want to support some instead of hand coding like a list of like okay you can use e with an accent and you can use an um lock because like motley crew but like we're actually not ever going to use an upside down question mark instead of doing that what we ended up doing is just using our production data we had a bounded set of search terms um because we had a bounded set of things we had content on so we just pulled them from production and we loaded them into memory and we actually used a histogram to represent how frequently they were searched for it and we got a pretty accurate um coverage of our code you also could just do it randomly you could even that you could even if you wanted to reverse the histogram and try to like cover the ones that don't get searched that often more often that's pretty cool um a co-worker of mine took that idea a step farther and he actually built took a corpus of all the english words used in all the content we had and a histogram of that and put that into skull check so he could generate english e-sounding documents which were useful for testing machine learning stuff on he used those for some scholarship tests but actually you have a method in scholar check scholarship's a pretty small library right so it generates all these inputs you can tell it how to generate a document using this frequency histogram that you've defined and then you can pull samples from it there's a it's literally called sample it's dot sample and it gives you a a piece of data that's defined by your generator you can use that in anything um he was he used it as like sort of a base function for some machine learning code he did uh we've i've also used it for performance testing have you guys ever used scala meter it's a micro benchmarking framework that uses this dsl that's really pretty similar to scala check so it was a useful thing to do to say i already have generators for this if i want to see how fast a particular algorithm performs on it i can just convert those pretty easily into a scala meter generator it's a little bit you have to be a little bit careful because scala metered on purpose runs over static values because otherwise your your performance benchmarks aren't that useful and so if you're biased too much towards towards edge cases it's not that that's not a useful performance number it's just like a really specific performance number but that's a two examples of using scala check in a non-test way just for the generators so those are generators um the other the second thing i said i was gonna cover is running the scala check test so there's a bunch of ways you can do it one scholarship has its own runner um and own sort of syntax for tests that's pretty simple it's like basically assertion based if you're already using a testing framework like scholar test or specs to there's integration for that you also can run it on the command line every property that you define has a built-in main method which is pretty cool because you can integrate that into like custom build processes there's a test call back you can do all sorts of rube goldberging machine things with your scala check tests um one cool example that i've seen recently is jessica kerr has an example using scala test selenium integration with scala check integration so she's basically controlling the browser with her mind like you don't need actual monkeys at typewriters to make random input go into the browser but it's kind of cool because selenium lets you execute javascript from scala so you can like test your you can properly you can use property you can basically write property based tests of your javascript without writing any more javascript and knowing scholar programmers i think that's something you guys would be excited to hear about uh yeah so that's a way you can run it when you write the test they actually end up being fairly similar to how you write example-based tests um if you have sort of a feeling for how to write unit tests you're gonna sort of which involves often sort of gravitating towards edge cases because that's where our code tends to break you're going to be good at writing scala check tests there is a little bit of a wrinkle in that um you want to write like a verification method that doesn't just duplicate the logic that you use in the actual code because otherwise like you're just testing you know sort of equality or your ability to remember how you wrote the original code one like useful principle that i've read about is doing round trip properties so if you have uh if you have methods that come in pairs that do opposing things to data really common thing would be encoding and decoding what you can do is do like a round trip test and say that the if you if you apply the encoding and then the decoding it equals the initial output input and that's really um that's kind of elegant because you get to test both like oftentimes there'll be bugs in the princess kiss that weren't in the turn into frog so that's kind of cool this is all covers basically everything that i've covered so far is is pretty functional right these are self-contained you have to be really careful with side effects because it runs these tests over and over and over again this is best suited for things that don't alter state scholarship has a a set of traits that's designed for testing stateful systems and i'm not gonna lie i don't totally understand it it's like kind of like mind exploding um it's basically like you make this abstract state model and then you like define commands that operate on it and then scala check will like create a sequence of commands like random sequences of commands and you've defined post conditions and it checks that the post conditions always hold and you can define paths that it can take and pass that it can't take it's pretty cool um i encourage you to check out this presentation by ricky nielsen who wrote the scala check library and maintains it he's using nix os and scala check to generate vms with like randomly generated properties like memory and os version and ip address and it like gens the configuration the test spins them up and then it uses these commands to see what happens when they ping each other in different it's it's like whoa whoa it's it's crazy town it's honestly kind of edging into skynet for me and i can't read about it too much like right before bed but definitely check that out it's pretty cool that's some things you can do with writing of the tests so when you run them there are certain properties where you can test like the total set of input those aren't very interesting usually right it's like an enum test or something most of the time you're going to be constrained by the fact that it takes time to run tests you have to have some bounds and that's where params for running tests kind of come in this is the ones that you can set up and you can set these up in code or you can often set them up in the runner depends on how you're running them minimum successful tests uh is it doesn't just run it once it runs these it runs these properties a bunch of times over a bunch of input it's defaults to 100 it's going to test each property that you find at least that number of times unless it finds a false case and if it finds a false case it's going to stop generating new ones according to your generator rules and then it's going to go ahead and try to shrink them if you increase this you're going to increase safety right you're going to have more test coverage the more minimum successful test you have but it takes longer to run so you know something you could do is on your jenkins builds late at night maybe increase the size more and run an hour long thing but then when you're just doing build integration run a shorter one there's things you can do there max discarded has to do with that uh filtering method right sometimes you're you've written code that has like pretty dis like constrained inputs and it'll give up it'll be like i can't i don't what do you want from me nothing's passing uh sometimes you have to increase this although it's kind of a smell like hopefully you can generate things in such a way that it's not just making things and discarding them because that's a waste of time and min size and max size so when it's testing your code it's going to start with the the generator it's going to give it a size it's going to start with like a the minimum size and it's going to increase it up to max size some generators don't care if you're picking one of five things it doesn't matter what size you get but if you want a list of things and it's an arbitrarily sized list it's going to start small and build up so that's pretty important especially if you're storing stuff in memory you can run into issues with this um it makes the most sense for collections and you can use this yourself with uh the size fram four gens the last one is number of worker worker threads like i said you want to be careful about side effects cause if it's not one it will run them in parallel and you may find race conditions which is good but um you want to be aware of what you're doing so that's running and then the very last thing is shrinking and this to me again is a little bit sky nutty it's crazy to me it knows the mistake i made before i did it's pretty cool when it finds a failure in your code it stops jenning and it moves into shrinking and it looks for a shrink instance for the gen types which is going to like shrink down that space to try like in literally if it finds a failure int it's going to literally make it like smaller basically with a tuple it'll go for each one it'll try to first shrink the first one then the second one then the third one with a list it'll just like remove chunks until it gets it to stop failing um by default if it doesn't have a shrink and there's a bunch of shrinks provided but if it can't find a shrink and this is common if it's a user-defined class it'll it won't shrink at all and it's still in a mentally useful library but you can define your own shrinks if you want to especially if you're doing custom data like uh data structures that's probably something that might be useful to you uh this is actually just the actual shrink default shrink implementation for in but i witched it up a little bit um as you can see what it does is it takes a failure and it needs to return a stream so it's going to go in and it's going to take the failing one and like go down by halves all the way down and then it's going to interleave positive and negative to try to shrink to be like which insert really failing so that's pretty cool okay that was a whirlwind tour of cool stuff in scala check now i'm going to attempt to turn it over to my dear colleague stu who is not actually here as you may have noticed he's not a ghost he's uh driving through north carolina yeah one of those states um on you know urgent family business but he has done us the good deed of recording a pretty cool little tour of how all this stuff fits together this isn't technically live coding since he recorded it but knowing stu unless there's like video editing software in emacs he basically recorded it all in one go so this is as close to live coding as you're going to get and i'm going to try right now to play it with his voice but we don't have audio input so i'm going to try it with a microphone and if that doesn't work i'm going to put my headphones on and we'll like you end it do simultaneous translation but let's see if we can get stew's voice up on the thing let's call it check in a small project if you look at the build sbt i'm going to start with it does nothing but import the scholar check library we're going to create a single test we're going to call it updown.scala and it's going to be a scala check test it's going to have a single spec called up down and it's going to test a single property which is the up then down is the same as down it's going to take a string as input and it's going to do just that it's going to take the string convert it to uppercase then to lowercase and see if that's the same as taking the same string and just converting it to lowercase let's go to a shell and try to run our test i might expect that this test would pass but actually it will fail often and the reason it fails is that there are many unicode characters where two different lowercase characters share the same uppercase character so the interesting thing in the output here i wanted to point out is that we get two different values back when we get a failed test here we got arg0 original which is the original test case that failed and then what it's done is given us an arg0 which is just a single character which causes the problem for us it does that by taking our original string and shrinking it into smaller and smaller pieces so it's narrowed it down to a single failing character let's play around in the console and see how this works if i import org dot scala check i can play with things in implicit scope called shrink instances there are shrinkages for lots of different values if you look at the string instance we can ask it to shrink a string for us take asdf when i run shrink it's going to give me back a stream if i force the stream we can see all the things on the stream and starting from my asdf original string what it's done is it's given me some smaller versions of that same string to try as possible new test cases if we were to take one of these if one of those were to fail it would give me back even smaller and smaller cases until i get an empty string which when i ask it to shrink the empty string it's going to tell me uh it's going to give me back an empty stream saying i can't shrink that any further and there are these shrink instances for lots of different types i can shrink you know ins for instance and i can even shrink tuples of string and int if i give it a a tuple of a string and an it will give me a stream of possible new values where it first tries to shrink the string and then it tries to shrink the end so let's go back to our test case um here uh is not the most interesting thing to be testing what's much more interesting to be tasting is if we had uh people because per people are more interesting than strings going to say that people have names and they have ages um and so we're going to change our test case to instead of taking strings to take people and now we're going to try and drink their names i don't mean shrink i mean convert them from uppercase to lowercase if i were to try to run my test case here it's going to fail on me saying that it doesn't know how to create arbitrary people yet so that's something we know how to do we can put an implicit scope a arbitrary person and we can create that using a four comprehension people will have arbitrary names and they'll have arbitrary ages and that is how you make people and now i expect that when i try and run my test case again we'll know how to create arbitrary people and therefore create arbitrary test cases and i'll find failing test cases however in this case it no longer is giving us back a shrunk version of their name i get this long string and it's going to be tough for me to figure out why this big string failed that's because it does not know how to shrink people because that's something i just made up so we can fix that by uh telling you how to shrink people by creating an implicit person shrinker which is going to be of type shrink person now i'm going to use a constructor they give us that takes a person as input and returns that stream of shrunken people um but that's not going to be too easy to create but on my own but knowing that i have a way of shrinking tuples we should be able to shrink people if we know how to shrink tuples of string and it now how would we do that so first we'll ask it to shrink a person but it can't strike a person it knows how to shrink a tuple but i can take a trade a tuple that knows how to shrink out of this by tupling the person by name and by age now this is going to give me back a stream of tuples but i need a stream of people so i can map the stream and we can use the person apply method that comes with our case classes as long as i tuple it as well so now i expect that when i try to run my test case again i will get failing tests and now the name is shrunk for me again this is great now i have much easier failing test cases to work with however this is not fun to write this is something that's a little bit tedious and it might be tedious to remember how exactly we did it and so you'd hope that maybe there's some magic to make this happen automatically and as it turns out there is and whenever you're talking about magic that involves case classes and tuples some of you may guess where we're gonna go with this and that magic often comes from the shapeless library in this case we're going to use escape shapeless scala check which is part of the shapeless contrib project and it provides exactly the magic we're looking for in this case so all i'm going to have to do here is import shapeless.contrib dot scala check and now if we reload our project and make sure that it still does compile what i'll be able to find is that now i don't need to create these shrink instances and in fact shapeless will know how to just put them into explicit scope for me and in fact i can get rid of the uh arbitrary instance i had to create shaples will know how to create that for me as well and we'll test it creates arbitrary people which are arbitrarily shrunk so that's great and if uh however it's not always going to be the case that you can use the arbitrary stuff that scala check creates for you you might have to create it yourself for instance we might say that people have birthdays and a birthday is going to take a person and return a new person and that's going to be by making a copy where the age is increased by one so we might test something like this and say that birth days make you older and we'll say that person after they've had a birthday their age is greater than the person we started with and if we run this we'll find that we actually get a failing test case because someone's whose age is max int when we increase it we actually get min int and that's not greater than accent so our test case is failing however this is really because this is a bad uh way of generating arbitrary ages for people and the point is that if we were to use the that's the same code that we're going to get from the shapeless library which will also fail so in cases like these what i'm have to do is i'm going to have to create the generators of myself which create more reasonable people and reasonable people only live to be 300 years old and now my test case will fail so the magic is nice when you can use it you won't be always be able to use it for a lot but for a lot of the cases that you're going to generate you'll find that scala check is able to do a lot of the stuff for you which makes writing these tests a lot more pleasant and a lot easier yeah so that's a little bit of magic um oh wow i'm back in the real world now um that was hard i don't know how they do it at the un that was just english to english too um so yeah i'm gonna wind it up uh we have to a little bit time for questions but besides questions because this is usually what they are anyway if anyone who is doing scholarship is doing something cool with it and would like to share that would be awesome i would love to hear about it so yeah anybody