Property-Based Testing in ScalaTest 3.1

SF Scala: Bill Venners, Property-based Testing in ScalaTest 3.1

Recording: SF Scala: Bill Venners, Property-based Testing in ScalaTest 3.1

[Music] so I'm going to explain property based testing in general and show how things would uninstall check and what is coming in Scala test and I would it's not actually finished the stuff we're doing in Scala tests so I would like to get your feedback if there's some way that we were thinking about doing it that you don't like or you think would be better but I'd love to hear about it and so maybe you can actually influence the design a little bit but just to give you a kind of some basis on why I think it's important I wanted to start with this page of code does anybody know what language that is not Scala well see that's what happened Eiffel didn't make it in the marketplace this is Eiffel code I was done by a guy named Bertrand Meyer and this was kind of competitor to C++ and C++ one out in those days and one of the ideas he had in Eiffel is called design by contract he trademarked designed by contract TM and in I thought you actually had to specify a function when you wrote the function so there's these three parts to a function body this is the square root function real is like double right and you have to say require and then you put some assertions that are called preconditions then there's du which is actually the body because the back in those days you're actually usually doing side effects so it is do kind of make sense and then there's insurer City sure ensure where you put what assertions he called post conditions so preconditions the contract was preconditions are the the responsibility of the caller and post conditions are the responsibility of the provider of the service the person who writes the function so the deal is if you give me good input I'll give you a good output right that's the contract essentially but it was a way to specify what do is doing right so what he says here in this case is a square root function that'll take any real number and if it's 0 or greater are positive or 0 then it's gonna trick come how compute the square root I left that at dot dot that's just an exercise for you guys later if you want to do that an eyeful then the way the post condition works is he takes the result that would there's sort of candid the results of return which is the square root of x square it and in the ideal world you'd get X back right but this is not the ideal world it's real or double right so it there may be a little rounding error so what he does is he takes the absolute value well he subtracts X actually that gives you the ideal world zero so then he makes that little difference possible difference positive by taking the absolute value and then he lets it be make sure it's less than a tolerance he's going to it to to admit so Scala actually has two things in it that support this style is designed by contract so I'll assume how many people news go ahead and assume in pre diff one so nobody knows it it's the it isn't a Scala book that the share way book in a footnote in really tiny print it's there it does exact same thing as assert but it's intended for preconditions so it's what it does is it throws assertion error like on the JVM that's what it grows then there's something called insuring and so this is like a you know I written a square root all spelled out method here and my assumed just says it has to be greater than equal to zero and it's not positive infinity and then then I call math dot square what Matt thought square root does is if it's if it's a negative number it gives you back not a number so what my function that is going to do differently is it'll throw an exception if it's a negative number instead of giving back not a number so what my ensuring Clause does the same thing the other the Eiffel code did it squares the results the when you call like the candidate result is returned by Mathis square root X so on that I invoke ensuring and I pass to ensuring a function that switched into which that result is passed and then I compute my tolerance I square my candidate results subtract X in the idea world that's 0 right compute the absolute value make sure it's positive as I compare to get star so it's the same thing so essentially people don't mostly do this this didn't win in the marketplace of ideas what one of the marketplace of ideas with j-unit right you take the post conditions and you put them in this other place so you'll put a mix it in you with your body your main code you put in this other place called a test so the precondition though yes it's in preed if you can call it on anything so I'll show you an exhibit it's very simple let's say console here this is SBT let me make that a little bigger see what that looks like that's too small Oh test : console I'll just show you test oh alright test : console so basically I decided since this isn't all finished that the talk should mostly be demos so it's mostly demos but if I just say 42 just as a random number ensuring and then what you do is you have a method coming like you pass to it a meth that will into which 42 will be passed and you can say I greater than zero that will be true it just returns 42 so that gets returned but if say less than zero then it throws assertion error so it's an assertion and nobody uses it very very rarely do I find people using that not reading people even though it's there what we use is this we use require which was the JVM way I mean actually one of the things that that Eiffel did was it made very fine-grain ways to turn on and off assertions so the idea was if I go back to the Eiffel code here I write all this but it all necessarily want to run that all the time so it may be slow especially in those days this is 20 years ago or whatever so by default what is enabled is just the require clauses those are run by default and then the default is the ensuring clauses are not run so you would like enable them for development and testing and then you turn them off or four or even whatever you want to turn off so what we what the Java people settled on is we're going to have these require precondition checks baked in but we don't throw assertion here because the assertion air can be turned off on the JVM right we throw something that be turned off illegal argument exception or illegal state exception are no point exception those are the ones that we threw right so we did this and we took the you know the post condition and we stuck it in a test so this test is a is written in terms of a property-based test that says for all doubles whenever the precondition is met then the post condition should be met and so I have the add that to move it to a test I actually have to add that for all because we go back and look at the look at the Scala code here basically that assume and this ensuring they're going to be called on any X that can be passed in right they have to be true for any kind of X that can be passed in so all X's essentially so that's that's what you can say in in Scala tests currently with Scala check behind the scenes running the the property check and it's called a property the property is that you know the result F you square it should be equal to the original within a tolerance right that's the property that all that that function should behave or should exhibit okay so so anyway that means we actually repeat the precondition twice we say it once in the test and we say it once in the production code it's kind of duplicate but the reason I thought this was important is because I think this I have this one talk I gave a couple times called the essence of testing and I said that the essence of testing is this kind of logical formula which says for all X px implies Q X so it's kind of what you what happens if you take design by contract which has these things inside the body and you move it out to somewhere else you end up with a statement of logic a logical statement which specifies the function that you're testing and and so in our case when I said for all x : double that's the upside-down ax and then p is the precondition have the goal p that is that it has to be greater than equal to zero and it's not infinite right and then Q F is like the the function under test so that's like in our case square root spelled out X is still X but then you cast in f of X you get the answer so basically I make an assertion using the function f and the original thing X so that's I think kind of important it's really fundamental to testing it's not this kind of thing off to the side that Haskell people did so I think it's important to Scala test to be have really good support for this and one of the things that we are doing with Scala tests in general is migrating our third-party integrations to modules because scale test is really too big and so one way to make it smaller is just move thing rename things and move them to separate modules so if you just ask for Scala tests you don't get the J unit at the testing G the selenium if you're not using that the Scala mock the vaquita the J mock all those things are kind of all bundled together and then also Scala checks we're just moving it out to a separate module so we'll just rename the packages but we'll work the same but one of the reasons that we want to do that for a Scala check in particular is because of when a new version of Scala comes out Scala test has got is one of the first it's like really in front of a lot of things to be able to be built because why do people use it in their project so we use Scala check in Scala XML that's the only two things we use so we have to wait for scholar check to be built before we can build scale tests for a new version of Scala like the skull of milestone releases and whatnot so back in - mmm five that actually took like six weeks before Scala check was working for that one and then after we got it we in a day we had and that was not even working yet for a scholar J as just JVM so then within a day we were able to build it so we'd like to just once it's a module a separate module then we can just build it right away and then when it comes along it comes along but there were a lot of other things also that I thought could be done differently so I want to show you this is what I'm gonna go over during my talk and I'm gonna mostly show demos and that face demo all take some a few questions not a whole bunch but I'll limit it but just so you can if you have questions you can ask as you go and remind me to repeat the question if I forget so the first one is generators are available directly no arbitrary there's a little little minor but they'll kind of get my demo started import org dot skullet check so for some reason Ricard and Nielsen the fellow who did Scala check wrapped that you even with the generator to be the implicitly available thing you to grab it you need to say let's say Scala check ARB into our bonds making an interim equals implicitly arbitrary of end so you grab an arbitrary implicitly and I got one from Scala check so then I can say the generator is just called the arbiter arbitrary alright and now I got a generator oops I forgot to say Val okay now I got a generator so now I can with that guy I can say sample and you can see some hints this is an ant generator okay so you know it's made a difference but first scala tests generator what you say is just st antigen equals implicitly generator event and you just get one okay so then you can get samples out of it for example so you can see that basically they generate ants and let's see what okay so another thing that you may have noticed is that basically when you ask a Scala test generator for name to get an ant but yes it's call a check you get a you get an option of an int so that was another thing that that basically I thought was it could be a would be better if a generator of T just gave you T's gave you served you tea rather than maybe D so the reason that that it does that is just so that he can do things like this there's something called such that so if I take my Scala check it's a generator and I say such that I can say given a generated I let's only how do I do that let's only now I don't remember what such that takes let's only oh it's bully and that's what it takes so let's only do positive numbers or you know one and running greater and now I've got a new one well let's name it something Val pause aunt Jen I'll call it skull of chicks palsy and Jin like that so that's a nice one liner and then when I use that guy and get samples I get some nuns anything that would have been negative or zero comes out none right so so in Scala test and it's got a check you could also do this you normally compose new generators of old ones with a for-expression so I'll make one called Scala check pause engine sorry did that already spell with this applause engine like this and I'll just say four and you say for every for an integer in the the the generator I was call it test pot int Jen so that was the one I grabbed implicitly right I have a generator Vince I'm gonna take an inch out of that and then I'm just gonna say J is equal to what I want to if it's greater than zero that's cool I'll just return it else I want to do something else I could just return 42 or I could take the absolute value of it right I'll say I'd at ABS and then you just say yield J and now I've got a this is a generator event that will only generate positive numbers with no sums or nuns it's just all we ever eat just give you back another positive number so Zen it can anybody see a bug in my my slide is there hmm I mean assuming that that my I'm promising you I'll give you a positive number is there any way this could give a non positive number I know I equals zero we'll get oh yeah it'll give zero you're right that's one okay there's there's there's a bug I didn't know about men yes what happens with min value ABS yeah so in time in value is one bigger than int duck max value that's what the problem is and it's kind of good to know see if there's an eight and a seven so when you take in 10 min value duck abs you get back in 10 min value so that's another way I can there's two bugs in this one so I'm gonna say s T paws into Jin dot sample anyone a buddy want to place any money I'll get either zero or minute no good smart this is like two to the 32 to get either one of those because we try to let's see if we get it Oh it'll be a while okay let's give up on it so but there is a bug in that one but that's basically what you're forced to do you just have to like if you want to you know get rid of something you have to replace it with something else and you've got an inch in right okay so that's one slight difference and then another one is repeatability this is something that since we started our work which we did a couple years ago it got fixed in Scala check so they actually broke some code in Scala check to add add this so I'm gonna actually run this flaky test spec so kind of on the same well can anybody tell me why this test would be flaky so this says for all make a little bigger for all in X plus 1 should be greater than X S Plus is a I'm testing Plus on ant now just make sure it works anybody huh MaxEnt right because it overflows that'll be negative so so hopefully when I run this thing let me say : Q and I'll say test only star flaky test spec I should get some failures in some that's a success that's a failure that's a success that's a success that's a failure so it comes out fairly often the reason is we try edge cases and Max values in edge case but it's hard to reproduce so what what the feature that that people wanted was that I could actually reproduce one of these failures exactly so we print out the seed essentially what scholar check originally did was they used Javas random number generator which is just mutable so what you have to do to be able to make it reproducible is have a functional random number generator that is in it's a immutable object and it'll generate the next int and give you back at the next random number generator that you feed through right so if you just give that same the same seed then I can reproduce that error the same every time so then to go fix it one way we could fix it is I can sigil we're never in here whenever X is not equal to X value and I'll say this assertion so I did not specify the whole behavior because it might be it's probably better to say if it's ain't got max value then the answer is whatever it is min value otherwise it's this thing but just to show you this this so now because I'm running with the exact same data I should know that I fixed it right so I won't just show you again this this this shape I've got for all in X's int now the precondition the P is X is not equal to max value right and the post condition is when you add one you call the plus method and pass in one the results should be greater than the original that's my cue and f in this case the function I'm calling I'm testing is plus so okay okay so that's that's a demo anybody have oh yes you have question perfect timing yes I feel like that I just reminded of the the body Python where they asked him about the swallow and he gets ejected I don't know that's a good question so what I can say is that each each for all gets the seed so when when you start up Scala tests either specify seed and it gets stuck in a variable that's like global it's in it's in the randomizer companion object and randomizer is the immutable random number generator if you don't do that then there's one that's the first time it's used it just snaps to something grabbed from the time of you know right so and but it's each like if you have ten different tests with ten different for alls they each get that number they start with it so each test starts with that each per all starts with that yeah so so that means every for all is using the same seed but that's okay because the next time it'll be a different seed and it doesn't matter really so if you have parallel what in Scala test normally happens is sweets run in parallel but it within a sweet the default is the tests are not running apparel you have to make some parallel tests execution but then each test is a sequential thing so I think you'd it's basically everybody's gonna be using that same initial seed it would be the same seed it should work so in any other question I'll take two each time okay okay so let's see here go back to my slide so that's randomizer and flaky test spec and just just to show you randomizer in the Scala talk here see how that's really tiny make it bigger it's in it's in the prop or will be in 3.1 in the profit package randomizer and it's just it's immutable class you know it's a functional object that has a bunch of methods here's choosing things between a min and a max and then there's just a bunch of next methods next off of different kinds of types but you don't just get that back you also get the next randomizer because this thing's immutable so for it to be different next time you need to use the next one the next randomizer so you can see that right and these outputs there they always outputs a tuple with an ant and a randomizer or a double in a randomizer so that's how that works and I think Scala check has that now too so so what is going on the way these are implemented is with implicit so you saw me looking up implicit so what the for all the way the for all works is you pass a function to it it has a type for its input parameter or parameters right here I just have one parameter to type int so there's this hidden implicit parameter being passed at the end and it's going to sit you know Scala compiler is gonna look up for an implicit ant generator like I just did by my hand by hand um it's called intz so it fills it in there that's where it comes from so that's how for all knows how to get that values of that type and that's kind of like one of the big ideas of property based testing is you don't have to think about like okay though I tried this one I try this one I try this somebody choosing my hand it's just you you have to think about well how do I get a generator for the type I'm testing and then let it generate them and and you hopefully will cover more bases so if it's double like this one then there was a little doubles it's called doubles and that's an implicit generator double that's just grabbed and used by the for all so that's how that works and so just to show you that I'll go back to test : console and I'm just saying it so it's called int that's a generator int and I can say samples a hundred so you can just see some of them or double samples 100 basically the general you know generators generate whoops whoops I can spell doubles okay and so one difference with Scala check is that that like our samples you see they're evenly distributed where spell check will mix in edge conditions into the samples but but that's basically just to you know that's what these things do they generate data of some type and so you know for lots of types that you might use there's just off-the-shelf generators but that you need to be able to make your own so what you usually do with that is a for expression like I showed before so another way to do this thing would be instead of adding it or whatever clause is to use a generator that doesn't produce what was it max int max value into a max value so let's actually change this back to one that flip fails make it fail again pump go back to this guy whoops well that yeah that definitely that that one I know will fail so I went I got just a fail again because I'm using that same seed so what I can do to make a generator is I'll say Val in six other than except for max value I'll call it for the heck of it and then I'll say four and again I'll take an eye out of the regular one which is intz and then what I'll do is say J equals if J equals in max value let's use 42 why not else J I sorry that's not gonna work that's recursive if I write is that right why is it red oh because I haven't put a yield yield J so I've actually needed it so I can I can make this implicit and it should actually that should heal it because now this one's gonna overpower it's gonna find this one in scope instead of this is well let's have IntelliJ tell us what it is I can actually wait feel light bulb and say add type annotation it is a generator event right so we just made a generator event because we started with a generator event and we yielded an int so it's gonna find that one this should actually seal it go here okay so that did just want to show a few things and you can do this with Scala check as well if you have a custom generator instead of making it implicit you can just pass it in explicitly but not at the end you can just say for all its accept max value I'm gonna get rid of that for I said for and I typed it okay so I can do that and that's another way to do it that should work you can just pass it in there so you can say for all let's get max Val to make it more obvious I think it's more obvious you can also pass in if you notice here the name of the parameter is X oh it's gotta get to fail again let's get it to fail by getting rid of this so when it fails it just gives you the the value that was passed in here you can actually if you want put the name of the parameter here like this it's called it you call it Fred or whatever and then that gets that'll get reported in the error message I said that that the X parameter that helps when there's like five parameters but they're in order and then if you wanna you can combine these things and put this put them together I think I need two parentheses here because that's a tuple but what thank you IntelliJ okay and now how did I make it oh I made it pass because I put the generator that doesn't generate MaxEnt back in so those are some things you do all those things let's call it check integration with skull test as well but that's kind of the ideas you make uh you make a new Jenner you make new two generators how two old ones yes this is Scott this will be spelt as 3.1 and we're close we have a first pass of all the documentation finished so we need to edit that and maybe few more passes we don't have a few pieces that I noticed actually preparing this talk oops that's not there yet so I have to write that but it's it's not that far off yes yeah so for case classes what you you could do there's a shorthand for it but I let me show you the what you could do is let's say case class person named Colin string age pulling it now there's you know this will allow negative ages and empty names and that sort of thing so you might want to but we'll pretend like that's fine for now so I'll say person gin to make a person gin I would say for take the age out of the int we have posits two but well let's do that why not pause and two values it could be zero so Posey and value so basically there's these off-the-shelf that's an a Posey is named after Buster Posey I'm sitting in San Francisco figured somebody would know who that is named in how I usually put name first but it doesn't matter so name let's say I'll just make it out of strings it's gonna have a Chinese name most likely because there's a lot more of those than there are letters we could say often numeric strings or alpha strings or whatever but I'll say yield person come at age so now I can say person gentles let's say 20 dot for each print line you'll just see them lined up see what they're they look like whoops okay we have some yeah I told you it's mostly the Chinese are taking over and look at how old this Chinese that it's got to be a wise old Chinese person they're like yeah so that's that's it's this thing normally you make a 4 like this and you just take some generators and you yield a new one so it's a monad any other questions about that I did not they do that's correct yes that's one of mine it might be my next one let me see if that's my next topic yes very good a very good segue so what what he said was that scale checks put some emphasis on edge cases so I I said favors but puts emphasis is a good one so let me let's see I mean grab we have an SC let's try this guy so what you can do in Scala check they don't have like a way to get a 100 samples you just got to collect them so I don't have we called the int generator st SC engine ok you may not be in scope anymore because I closed it ok I'll have to make it again so implicitly actually have the code right here I'll just use this one copy and paste so there's my import this is stuff we did before I got an arbitrary I just called it art this time got a gin I called a gin okay this time and then now you know if I say gin dot sample there's my there's my thing and you can see you'll see right there there's an HK see comin L there's another one so let's get a thousand of them and what I'm gonna do with this so thousand Lestat fill will fill a list to a thousand elements when I say and the second thing is the the function it calls for each elements it's going to call that function a thousand times going to get a thousand samples and because I have faith in Scala checks and generator I believe it will always be a sum I think it is but it's basically faith so I say doc yet right and that's that's how I get the actual a list of a bunch of intz so what I'm going to do with those is try to count them so the first thing I'll do is I'll group them so this guy says take that thousand ants and group it by the int itself so now you'll see for example that this number appeared once in the list because it's associated with a list of itself one time but this zero here appears many times in the list it's associated with a list of all the times it appeared in the thousand ants right because they favor edge cases so then what I want to try to do is just get rid of the ones that that I'm gonna filter out exclude the ones that have a length of 1 so filter means keep like retain the ones that pass this function so the ones that don't have one get to stay so I see two education is here zeros and and minute right and then what I'll do is if I say keys that's gonna give me the edge cases i spelt it wrong filtered there we go that's them so that's the same edge cases we use it's minute max and 0 minus 1 plus 1 minus 1 and plus 1 are just kind of throw it in for good measure of it so there's five of them that makes sense so let's get the length so what I'm doing in this one is I'm taking the fill the one that just has these five key value pairs and I'm changing the value instead of a list of you know a whole bunch of things just the length of that list because I don't know how many it made so now if I if I take that and print it you'll see about how many they generated so 92 111 93 96 96 so I think the heuristic is about half of them are edge cases so if I compute the some lengths dot values that's some it should be around 500 ok 488 sometimes it's above sometimes below essentially there's only five edge cases I tried them for under eighty eight times that's actually 483 times I've tried the same thing over and it's gonna do the same thing right so that was what I didn't like now if you compose two generators together and you have a tuple two with two minutes now there's not as many duplications right but I want to try to improve on that and because you know it's greenhouse gases why Bitcoin mining and this is like it's heating up the planet so what yeah it's just you think you're testing a lot but you might not be testing as much as you you think you are so what we do is different and what I thought I'd try is first we try edges and some of them some number and I try 20 percent at most or until we run out and then then we just evenly try evenly distributed values that's what we do and the other difference well actually no that's not a difference nevermind so let's try that so one thing you can do for example in this is now against back to Scala test you can see I can ask the ants generator for give me up to 20 edges and it'll you see the same fiver there it's the first list there and then it gives you back a randomizer so I'm passing it a randomizer so they can each time I get edges it'll put them in some random order for the heck of it because I don't always necessarily do all of them that's actually a list so doubles again this one is gotten a few more there's infinite negative any minus 1 this minus one ba-ba-ba-ba that's the smallest non zero let's see what is that now why is that before - wonder oh I should put these in a bet in the other order wait a minute no no that's the smallest that's the closest to infinity negative 1 that's it then minus 1.0 this is close to 0 negative 1 then negative 0 in doubles then a 0 that's right see that that's the whole point like one of the services the property based testing tries to provide is make you try to case you didn't wouldn't think of right there in the in the doubles generator so doubles is it I did yeah and there's one item one of the things I wanted to ask you about is one thing so everything that I've said so far as is also true in the positive side but there's one thing missing that might be an issue case can anybody think of Yes Man yes not a number you know yes oh yeah that's good what about pie but anyway I could I could go crazy but I'm not a number is one that I think we tend to not think about but it's overflow assume the thing is like adding one to MaxEnt so I think it should be in here does that what do you think yes okay I have one we can vote this is democracy one two I have a couple thumbs up you have taught your thumb okay you have people like that okay that's right and there's a way to exclude it finite floats instead of saying for all floats you say for all finite floats so I think float because it's float it's like for all floats means not a number as a float so and that's kind of the point is to make you think of that but the thing is not a numbers are pain to work with because equals doesn't work on it so you have to treat it specially but anyway did you have another comment oh yes mine that one that's right that's right I read that recently and I don't remember but yeah something simple like one point one doesn't actually yeah when you make it into binary it goes forever in it you have to yeah it's not precise I mean we can actually almost try it but anyway well I think it it's something simple like that at least okay so what we do if you look at there's length there's a size a misspell a check that goes up and down and then it just goes all of the Unicode area but so it just because there's more idea grams than there are letters you see mostly Chinese things and seems like which probably won't show up in your app but it might find a bug that's useful to fix right so here what I did is I printed it out what gets generated so the for all succeeds but it prints what it got so you can see there's two edges and that's 20% and it's different every time and the and the other thing is by default and this is true for it's called check integration in Scala test2 as a few years ago we were running them 100 by default because quick chick did 100 by default in Haskell Scala check did 100 by default in Scala and I didn't hunter by default in Scala test because Scala check did it because Haskell did it and like and what people do is they they write this privately based test and maybe it finds a bug most time it doesn't but sometimes it might and then you fix the bug and then you run it for the next 10 years and condition aggression and it never fails so why are we trying it a hundred times every time so I'd made the default 10 which you can just configure it to something else if you want more but I just felt like again it's the greenhouse gas issue I'm running tests that are never going to fail too much because I'll just you know within a day we'll have tried all the edges and you know so [Music] yeah yes yeah yeah I think I just did because I got a whole bunch more here so one thing you can do like hardcoded say give me at least a hundred minimum successful so I actually answer your question before you asked it that's like property based testing man so what this one looks like so that's one thing I added is you can you can add some config parameters right there and then there's an implicit configuration you can just set everywhere and mix it into every test if you want like always 100 100 100 42 or whatever your number is so what this one does is it takes two ends so it's the same implicit in generator that I'm grabbing then I print out you know the two values and then I again I just succeed so you can see them so here once you start combining them now there's a lot more going on here I get out of 100 I get 20 edges and then the rest are stranded so that's what we try to do differently and then there is one way - how do I get a hundred out of the first one this is if if I just do the first one but I do it a hundred times so member we were doing twenty percent so was - it actually we run out of edges after five and then we just switched to this even distribution so that's what I try not to you know I try to do the edges first but if there's too many for twenty percent I just grab random of those and do those so that each time you run it and the continues integration you get different edges twenty percent but if you run out then you just do the other guys so that's edges so that's different and yeah I think that's good so then so there's another thing called shrinking which is really cosmetic this is when you find like let's say there's a string that fails and it's 200 miles long and it's in sanskrit what it'll try to do is find a simpler string but for some i mean to humans it'll try to shrink it down to a smaller string or a simpler string so it'll try alphanumeric air to see if that fails and if they can find one within a reason about of time it'll use that instead in the test failure because then it's easier to read if I can't it just gives you the original that's all it is so in Scala check that was separate and this caused a rare bug but it was it hit me once you know there was this this example I grabbed from Stack Overflow so in you can see that he's using a record is using a stream still looks like and the way it works is like if I let's say I find 10 as my failure that actually doesn't work can you strength 10 and this will just give me smaller progressively smaller ones until you hit the end it which is called empty that's a stream is called a stream which I think was deprecated and dotty if if a hundred was the one that I found then it'll give me forget release Mahler once right so it starts with what I found it goes smaller excuse me so notice that in each of these cases I have a zero right before the empty so it does go to zero the shrinker so if I were to make a new generator I get if I have this I'll grab it again this is the my trusty anti generator from spell check I'm gonna drive this guy and then I'll explain the code this one is what I want to do is make that non zero and generator where I'm not passing in zero like for the square root maybe great I'm gonna make a function of the generators all zeros but I'm going to change the zero to one I should have made a 42 but that's that's similarly the code I show you for a Scala tests generally the same thing Chris Kyle Chucky you know there are monads you can make a for-expression to make a new one but when I make my for all and I'm gonna do a for all that fails actually wait a minute I will because I don't know how to actually pass it explicitly I'm gonna make it implicit by putting inside an arbitrary right because the generator can't be implicit Scala check it SP inside an arbor era carry and then I will make a property and this one is going to fail on first of all it divides I by I that needs equal one so that will work for anything except for 0 right that will blow up if it's a zero but I might generator that it's gonna pick up won't ever generate a zero so that's cool then I just make sure that I is less than a thousand that's going to fail as soon as I I'm sorry is I is less than a thousand yes so anytime it's bigger than a thousand it's going to fail which is going to happen pretty fast so when it fails let's say it fails with like MaxEnt then it's gonna try to shrink a MaxEnt and the problem is the shrinker doesn't know i'm not supposed to generate zeros so it throws in a zero and he blows up when during shrinking that's the that's the sort of inherent flaw with the having them separate so this is a when I check it when I actually check the property it oh this one didn't blow up did it yeah there does arithmetic divide by zero so it did it when it tried to shrink it it it blew up so what what we what we try to do in in in Scala tests was put them together so if I show the same kind of thing here I just need to make my generator implicit that's the Scala test one same generator won't generate zeros so it I get a failure but it doesn't blow up during shrinking but actually didn't shrink it so I think that's better so the shrinker is built into the generator now so it's actually one things when you compose new generators you get a new shrinker and usually I can't do too much with it so what I do is I just pick a few pretty numbers I call them Canonical's so for ants it's it's you can actually ask the instant generator Canonical's and then i have to pass in a round number generator to give it to sort it oh wait a second that comes back as an iterator and a randomizer that I'm gonna throw away it got to list now you can see them these are the Canonical's so that's actually that all I'm gonna use in my composed generator excuse me is that guy and that's good enough I mean it's just cosmetic if one of those happens to hit you know win the lotto and it actually causes the test to fail also then you'll get you'll get that as the output instead of the big number but what's nice is that if you ask the compose one where is it here how do you say give me your shrinks I have to say well it come out underscore equals let me go today what's this good this one so this guy doesn't generate zeros it also won't generate zeros win the shrink that's what it does it gets rid of them it actually turns it to a one which is redundant I'll show you I think I'll try to improve that but that's shrink and I say the number to shrink which was this big number we start with this one and I have to pass in a random number generator and I think I say it duck to list now you'll see what what was in the shrink list it's everything that was in the Canonical's but I got rid of two zero right so that that's another thing that we're trying to do different is when you compose your generator you couldn't get everything you get edges composed you get and and that can there's a confidence or erosion explosion problem there that could happen so it's all lazy it needs to be lazy so right now I think I'm returning a list which is not lazy so oh no no that that's right I know why return to list when you asked for the edges you give it how what's the maximum number to to to get and that's how why a return a list but it is lazy when we compute it so I'd say ten of them are 100 or whatever ten it won't give me more than that that's why it's a list so so let's see if I can demo that about this one this one I added a a I did actually get it straight to three because and just to show you that it will shrink I added this bit which is really I mean this is really contrived now it has to be less than a thousand and not equal to three so now three will cause it to fail which means it when you shrink and you try three it will say oh that's prettier than this big number right so anyway that's shrinking and Isis again that's just cosmetic it's not that important but it's it's nice that it doesn't blow up so the other thing that we did in 3.1 is add any bells to scholastic there's several out there now but now there's like ten times more there's a whole bunch more and these are Scala any bells which usually don't box but box when they need to box more often and we want but they they usually are just that the backing type so it's nonzero and is usually just an int at runtime like we hope but this one is is promising that it's not zero so I can say how that little works is nonzero and 42 well that's not zero so I get one and at runtime that's probably an end so I can I can do that but if I say zero at compile time I get an error so there's a macro that checks it at compile time for that it's valid so all of these have these same same things and then there's widening conversions between these and between the the regular primitive types and stuff so if I if I want oh wait before I get to the widening conversions I'll say Val x equals 42 Val y equals zero so if I cannot prove at compile time because those are literals I could tell a compile time that it was valid but if I can't prove at compile time then I say non zero and you'll get an error at compile time saying sorry you're gonna have to use a different factory method then apply so you can use from and you'll get an option so if I say from X that actually was valid so I got a sum if I say from why I got a none there is a way in 3.1 to say ensuring a valid and actually this is a assertion remember assuring is an assertion but this assertion changes the type so what you just you know if you have nearby evidence that something's true and you say ensuring valid it puts that knowledge into the type and now you don't have to worry not have to sir don't have to remember that that int is nonzero you actually have it in a type and there there turns out there was a lot of likes a my nearby evidence that X is going to be a valid nonzero grant up here I see not too far above in the code that I set at the 42 so if you ever accidentally do this then you get a certian error because it's an assertion essentially its assertion that once it's done that's now in the type and so then what is going to say is there are widening conversions so if you need an end and you have a nonzero int so let's say number R as 32 where as 32 is 42 so I can take that that 42 that that nonzero end and widen it to an ant and it just widens there's an implicit conversion because that's always safe right it'll never be a zero int but it's safe and you can even go the other way if it's a literal so if I have like a 42 that's actually an ant right that's type int so I say res 4 oh this is perfect res 42 you don't know how much I do practice is talk to you get that to work and just say non zero in now that's a narrowing conversion that may not succeed right but because of it's a macro whoops oh of course see I actually just helped myself because res 42 is not a literal so it can't prove a compile time that it's valid even though it has 42 in the name but if it actually is a 42 and it can prove at compile time that that is a valid narrowing conversion it'll do it and that's kind of like if you say Val be cool and they'll by calling byte equals 42 that will work because it you know 42 will fit in a byte right it also fit in a non 0 and so it's same kind of thing so what I did is is that's kind of the language of the generators because if you have a function that takes an int but you only want to pass it positive intz you can just say for all posits and that would be positive ends but when you pass the pause into something it takes an a just widens at the end so there are bike there's this is the this is the real end of my but the generators are in common generators right here you get these and scope when you have it mix in property checks so you get for example pause in so here's positives actually a generator of pause end but you can use it for intz if you want to test with ends but there's also one called pause and values that actually comes back is values that actually is generate event that's always positive to if you want if you just that's what you want right so there's tons of these generators based on these any bells and so kind of a social experiment is how do people to what extent will people use and how will they use any bells but I think one one use case for them that's pretty strong is property based testing in Scala this is to use those guys because you just say what you want a lot of times and and it's already there there's also yeah it's not essentially I had to take like four plus minus times divide had to actually put that on those any bells what would be nice is an export you know like that the int that you pass in they've talked about that in the past we can like instead of import it's the dual you export things from like the ant that is behind the non zero int there's an int there backing it export its its members and now they're the members of non zero int with one line of code what I had to do was write all of them so it's a lot of work doable but I mean it seems like that the language should make that easier so maybe someday we'll have exports but there's there's tuple generators function generators and then there's a bunch of between generators like so if non zero it's between two nonzero entry and and whatnot so there's a lots and lots of these guys okay so but any questions other questions so far good a function generator isn't too much different this was also something that Scala check that was one of things we wanted to try to solve the only generators that functions that were being generated scale check was originally constant functions but they did add cogent to spell check on all those you've seen what that is that's for generating interesting functions but what I just use I used hash code so it was a you know because it's there it's isn't Haskell we have hash code it was I thought simpler for people and the other thing I try to do in Scala is to make pretty error messages so if you're if you have a function like because of this function you know I'm saying for all functions from int to long if you map a list events the size will be well it's not gonna be four so I get it to fail right just to show you that the to string up the function actually shows something that you can copy and paste let's just say that's it that's just from the error message let me get that space there that's pretty then I can say F of 42 and I that's actually what I mean it's it's always this that was always going to give the same result when you pass the same thing but if I run the test again then it would be different next time so this function it's using it basically basing the output on the hashed hash code of the input because I always have that where is it right here so this function is slightly different because this is different that's the extra bit that's gonna be different each time G equals so G and F we're gonna return different values though so say F of 42 was this one G of 42 is that one so that's function generation so it's it's called check this this now but it uses something called cogent which i think is a little too it's a bit of unnecessary category theory but then this last last one is async properties Scala test has async tests so I would demo of that right here what an async test is to run this thing async fun sweet async property spec that's what it's called is test only star async properties spec is that what I called it I started them you know and use it like async property spec factory bean okay so that passed but what this guy is doing that's a little weird is ad soon takes zero too many ants and returns a future that adds them but also prints out what they were oh so so did I get them looks like it was one and two right that's because I hard-coded one and two in my test right here so I run it next time it's gonna be that again so what the way an async test works is this guy returns a future so if you have a future in a hand I have a future in hand called future some I can map an assertion on to it and return a future assertion back to Scala test instead of blocking in my test so I it's kind of like play you get a future of something they future database result or something and you map it and get a future response that you pass back to play and it puts that on the shelf and ask to be notified when it completes and when it completes it sends a response to the client what's called test does when it's given a future assertion is it puts on the shelf as to be notified when it's complete and when it completes it tells a reporter hey this test failed it was test exceeded yes that's true today yeah so this is nothing new oh yeah yeah sorry yeah so that's a sink in that and that's let's see so I'm gonna code come up with block comment and then I'll just undo the the property basement so one thing that people have wanted for a long time is to be able to do a for all in an async test and have that work and have the property returned if the preference of future assertion so what this one does it's really tiny is it it says for all at all this event for all this event when I call adds soon I pass it in that little underscore star means bust out the elements of a list and pass them in as varargs to ed ends it will add them all up and give me well not yet it will give me a future of that right a future some essentially then I map onto that feature some the same thing same kind of thing that it equals at the sum of the elements of the list I pass that I started with right but the result of that is a future assertion and that means your property is a future insertion so now in 3.1 that means this is a future the for all gives you back a future assertion which goes right back to skullet ISM and so it does work but what you'll see is that you'll get more things printed out you can see I tried a lot more interesting some things to add up here yeah I tried different no does not basically a test like the test body returns like this but there's a future that until that completes it won't if I put a sleep here then it's inside the future right the main one yeah yeah same leaks if I put a sleep here let's info it out but let's put a sleep here thread death geez let's sleep for a second each time maybe you could see that and it doesn't make you wait till too long see what this looks like should just take forever no I shouldn't this is compiled thank you okay where's it compiles oh this is like tread asleep oh come on there we go well no wait a second oh that's that's SBT my second he said I can't figure out how to show this what SPT does is it holds on to it until it's all there which is frustrated me because we do that and we can't get around their feature there's a way to turn it off but anyway you would see it spit out one a second at Italian that's what you would see if you believe me so I wait that's that's async and that's that's a that's some people in one for a long time any questions about async wait a little very very last one this one I'm just gonna like use a hand wavy thumb so going back to that original thing this is this guy if I can find it well I've lost it maybe I I closed it but anyway that that statement of logic there has been something that I don't know that we'll ever go out the door called the expectations which is a kind of assertion that as a result doesn't throw an exception and what what what this is is basically it looks it allows you to make logical statements in your test essentially so that's just like not gonna be in 3.1 may never be in Scala test I don't know but that was another thing too I mean what I'm sort of feeling like is that um you know this this thing that that I showed you here you know it's really fundamental I kind of want to say that functionally so so anyway that's all I got I have two quick announcements so basically one thing is we're doing these open public trainings called scala retreats which is you learn about skull in a beautiful place and that believe or not is the east bay not far from here so we're having fundamentals in advanced in march over there and effective scala down in southern california so if you're interested in that you can go to our keema combat training i also we are hiring too and we do consulting so we're available if you need help with scala and then i'm at the community rep on the advisory board of the scala center and and so i'm supposed to represent you to the Scala center which sort the open source at the nonprofit foundation that guides Scala and people don't use a complain so I started doing brown bag lunches for people and I'm happy to come into your company and do a brown bag of lunch for you and basically it's I'm kind of like a psychiatrist like ask you questions and you lay on the couch well the group and then I find out what you what I actually I show what's coming in Scala and try to figure out what people like what they don't like what they're worried about so so that's that's listening and that's all I got any other questions probably make them shorter so our shrinkers that was one of the things that I realized we didn't fill in all we filled in shrinkers for like int long float double but we didn't fill it in for non zero and so I actually wrote that when yesterday so I haven't we haven't like I said we're not finished so shrinking is one of the things I need to refresh my memory what we did and see if there was some way we can do more interesting shrinking out of composed generators I don't know if we can or not and I like I said it's kind of just for prettiness it's not that important if you know it's better that it just doesn't blow up library okay well well one thing that I forgot to mention is we go in the other direction so spellcheck goes from big to small we go from small to big and we'll say you know like there because when you compose these things there could be huge combinatorial explosions of them so that's done lazily but then we just try 100 or whatever we decide to try right so but let's just try it let's let's say a list of 1 2 3 Oh list dot range one to a hundred and then what what's what's the other thing we say here to get it to go print oh that's that's kind of hard to read let me put a space between it print can I say for each print line I'm just seeing what scholar check does and how he he gets rid of it but he's changing this 12 but negative 12 that's prettier this is fellow chick so I don't know actually I don't know but that doesn't look right I would think you would want to shorten them I would start with an empty list because we start small but any other questions yes [Music] well we want one for Scala test so that's well tuples tuple twos do I say an taient I don't know how I say that this is my own thing I don't know let's say samples 20 so does that look distributed I don't know yeah we need to test it so yeah there's some there's actually some classification stuff in here too that I didn't show you but skull check has that as well so does quick checks basically quick check was the founding father of all these things and there are other property based testing frameworks in the Scala ecosystem - they're just muck Scala check because again first mover advantage people mostly use that so yes no it's so check yeah okay oh I think well for ya I don't know I don't have options in either's so if if it's not there it looks like it is there all we will add that so that's again I was like discovering things that aren't there that I thought were there two years ago but anyway the other one that didn't show is I forgot that you asked about case buses normally what you like the full-blown one is to do four expressions the way I did that person one I had to know intz and you know how to know the names of those things so if you don't want to know the name so I'd say Cleese class I don't know dinner has me like a I don't know a drink and on string and a utensil a cost : int double thinking enough these days so if I if I I can say they'll dinner Jen I haven't had my dinner yet so I don't know why if that's why I'm thinking about this I need to say instances of dinner now that's all I wanted to say but I can't because I don't have a I need an apply and an unapplied that gives me back not an option but a thing and it turns out like yeah case classes actually have a none apply method returns an option that's always a some just like I believe I have faith that skull chicks and generate are always turns as I'm so I could have used it but it may not be right so I said you have to actually write a function that takes a dinner and gives you back the string d dot drink the string and d dot cost the double knot in an option I forgot there's an ass here I miss instances of dinner see if that worked so now that's another way to do it now I didn't have to say the implicit it picked up the implicit generator of double end string so if I say dinner Jin so there is one for that we have but it's unfortunate that this may be in you know spell three I'll be able to get rid of that because that's kind of ugly you know Jen dot samples let's get ten dinners oh that looks tasty especially this I love that I don't know what that is but yeah that's rig I like this price for dinner this negative 7 times you to minus 6 but not actually I don't like it because this is negative they don't know me much yes yeah so that's that was the let's see if I can remember what it is that's a great idea so here I cannot with instances of but you can you can do that in a for well Jen equals 4 and there was a cost in so I'll say what's the minimum will cost say doubles between I don't know 1.1 dollar and $100 or something I think that'll work we'll see oops it doesn't like that I need I lost my whole context here ok and then I need a name not a name but was the other one thank you drink in let's say alpha alpha what's it called I will fill it in for me well I'll just say strings whatever strings because I I think it's alpha strings but I don't want to screw it up yield and I'll say dinner drink was first cost was second and now we shouldn't get we should get some things between hundred ten in line let's put it out like this we can see it better but the names are the Greek names are really long oh there's a 267 right no no no this isn't right what did I do one dead oh and a hundred this isn't this is oh this is 33 oh yeah that's between 100 it's just about a huge number of decimal places so that though that's how you do that ranges yes yes yeah so one of the things that I don't think I've added yet it's both shrinking and so let me show you the API it's just you would have to make you can always make a subclass of generator that that there's a method on it that you would implement but I don't want you to have to do that I want you to be able to like use a for-expression to get a generator and say dot with edges and I don't think I've done that yet but with edges and then you put them in and then with shrinker and then you put in a function or something which we haven't done that yet but that I'd like that to be real simple but right now you could with this which isn't released by making a subclass but yeah so anybody else smell check is the same integration there's tons of code out there using Scala checks so all we're gonna do is is just with all these dependencies of external libraries is we'll rename them too or guess call this plus dot Scala check but you and then while give you a Scala fix tool to rewrite your code to when we release the 3 dot want to release that actually that the spell check one we're gonna change in 3.6 just to get it out there and then some sup we can release down the road we'll make that a module such that you have to add it to your build and if it's the same exact code so yeah most I mean it that won't change other than the name and that it's an external module okay we should probably quit as well at past eight and and people are thinking about dinner dinner between zero one and a hundred dollars okay thank you