Unveiling ScalaTest 3.0

sfscala.org: Bill Venners, Unveiling ScalaTest 3.0

Recording: sfscala.org: Bill Venners, Unveiling ScalaTest 3.0

hello I'm Bill and thank you for inviting me um what uh the grand plan that that Alexi and I hatched was that I would unveil the the new release of scalatest 3.0 but have you ever seen on uh uh TV or movies those kind of it's like you're running through this dark tunnel and there's light at the end but the farther you go the farther away that light gets that's kind of what this project has been like uh so it isn't finished and in fact uh there's some things I'll talk about today that I wouldn't mind getting some feedback on so it's kind of a could be a bit of a design review but uh there's a lot to talk about and uh I don't want to take uh take too much time because there's another speaker but what uh my challenge I felt was that there there's a lot of people using scholet tests uh it's pretty people are pretty happy with it um I would like to try to make it better but without screwing it up and uh I was mindful of a story that um I was told by Jim Waldo who created uh uh Genie is this one on by the way this is probably for that right okay um created a genie and at one point uh when I when I first saw Genie it was really beautifully simple and I I was quite attracted to it um and then uh at one point in its development um they added security and it it had no security allowed things to connect and and there was no way to like decide if you trusted that thing um and it was sold to the market as a way for things in the world to connect to each other so security was important it wasn't sold to the market as an Enterprise technology that would be behind a firewall where security wouldn't be needed because the firewall would provide the security so they solved it was a really hard problem and they solved it but it just sort of added so much complexity that it it kind of screwed up that Simplicity and he uh you know Genie didn't really work out in the market it just didn't happen and uh we were uh he's in Boston now he's a CTO of Harvard and me and uh Kim Arnold and him Ken Arnold designed Java spaces were having beer one night it was kind of like three old guys talking about why things didn't work out it's kind of depressing but um what he said was that he thought it was a mistake to do that because it to actually add that feature to solve that problem was a mistake and and because you lost the sort of the essence of Simplicity that it had and that's what I'm worried about with scholet test is it's already in 2.0 is pretty big and I like small things so I'm worried that you know what I do will make it worse somehow right so I'm going to give you an example where uh two summers ago I was already working on 3.0 and what I was going to try to solve with this the fact that one should equal one compiles when you know an INT and a string is compared for Quality that can never be true and it just wastes people's time that test will fail and then they fix it right so I did find a way and it was really hard because of of in matchers to combine matchers uh it made it very difficult to to do that you had to partially apply a type and that's not so easy in Scola but I figured out a way and that is actually in like the master branch of schola test if you check that out and build it and then go to the console and type that you'll get that error and it will tell you that it couldn't find evidence that string can equal int which is not too unreadable one thing I didn't like about it was that the strings on the right hand side and the ins on the left in the expression but in the air message it's reversed so that was I thought not nice but the real problem was that it uh took much longer to compile and so I checked it into Master saying well I'm going to make it faster and and for months I tried to make it faster and I just couldn't make it fast enough because there's so many of these there's so much test code out there and and all you do is write assertions right so I can't just say hey scal test 3.0 now three times slower to compile you know it just it's not a good it doesn't work so I I I just couldn't do it so I actually didn't do that and the last summer I tried a different approach was to make a compiler plugin that checks that stuff so that actually gives a nicer error message because I can make whatever error message I want and you can't tell that it's running I mean when you measure the compile time it actually was it has to be somewhat slower cuz it's doing more but it's not really perceptible we couldn't measure it and I think the reason is is because implicits are kind of a blunt instrument and whenever you put use implicits it has to search through all of them whereas a compiler plugin you can just go in with like a surgical instrument and and and find out what what you're looking for so so anyway we that's one thing I didn't do just because of that thing I think that might have screwed it up because there was just a lot more complexity and it slowed down the compile um so this is in it's in super safe which is a product but this stuff is free so there's like a like intelly has a Community Edition that's free that's there's a community edition of super safe that's free so all the scholet test assertions and equality like triple equals and and the shoulds and the musts are just you know uh now don't compile if you use that plugin so the main Beast that I've been wrestling with uh in the last year is SC .js and I don't uh you know mean that it's a beast but porting a 8-year-old library that was written for the jvm back when Scola I mean originally didn't have a lot of things like there was no future to Scola uh for example that didn't come into 2 10 right uh so we just we really used a lot of the jvm stuff um so for example we internationalized strings using the Java's inter string internationalization stuff and I thought I was being a good boy right you're supposed to do that and but it doesn't work on Scola JS so to Port scallet test to Scola JS I had to write a inter internation framework and that I think is sort of the uh definition of yak shave right so so I shav we shaved that yak and we didn't make a general one we just got ours working but what one nice thing that happened out of that is that before because we're using Java's uh string internationalization framework it was a runtime error if there was a non-existent string like you say I want to use this string and it's not there which is crazy so we did also solve that so now that that will that's a compiler error um so then another one was uh scholar check we the one scholar library that we integrate with is scholar check and that I try to keep dependencies to a minimum because it's easier for users um but scholar check is you know scholar check so I integrate with that and uh it didn't work on scholar JS um it I figured it would have eventually but we have scholar tests for in our own project testing scholar test so so there was another property test framework called Nia good name uh so we ported to that and and uh there's a guy in Malaysia who who helps with skullet test and and so he he was supposed to do that and because it said it worked right um so I got up the next morning he spent the whole day trying to get it to work but he finally did so I was like if he if he was going to spend a day get getting something to work he'd probably try to spend a day getting scholar Che to work on scholar JS uh so we used that for a while and then that didn't work and Scholar check still didn't work so I said I know we'll just write our own property based testing framework so that we can Port scholar test to scholar JS so we did that but just enough to just enough to get our stuff working right so it's again it's not a general thing um and then scol Che started working so we uh so then we went back to that so that's just two examples we thought it would take 9 days to Port skull test to scol JS that's what we figured and it's been over a year and it's still not done so um so anyway we've shaved several Yaks or this Yak several times and it doesn't look actually like it's too happy about having its picture taken even um but that's that's really been what has been holding up 3.0 and there's some other things that have nothing to do with scholar JS and 3.0 that have been kind of stuck behind this um so uh I want to just give you a couple uh simple slides of of of background uh so you know what what we started with uh this is um models this is an as that models the outcome of a test and so a test when you execute it in scholar test can either succeed fail cancel or or be pending and succeeded and pending are Singleton objects so when you if a test passes you don't actually want to know anything so you just get back a a Singleton um if uh pending it's again this test is pending that's just a Singleton but if it fails or canel people want to Stack Trace right so that actually holds on to a an exception canceled is always going to be test canel exception but failed is any throwable except sweet aborting exceptions like out of memory error there are certain errors that will actually cause the whole sweet the whole run to abort uh and not just a test to fail so that's what was added in 2.0 to model uh a test result and in um with fixture I in 2.0 changed the signature to have uh people deal with outcome so what with fixture is is it's a life cycle method on Suite that users can override to do something at the beginning and or end of a test and you can do all kinds of things because the test hasn't completed um when the test function Returns the with fixture has passed a a function from that takes no parameters and returns outcome so that is whatever you express as your test it gets wrapped into this function and passed into with fixure so you can manipulate it somehow and so the idiomatic well the the default implementation in Suite uh just invokes it so the job of with fixture is to invoke the test function so that one just invokes it the test function returns an outcome either succeeded fail canceled or pending and this with fixure just returns it back to run test which is calling with fixture um so users can over WR these in their suite and do something before and after for example so you can set up a fixture then invoke the test function in a try and then in finally you do clean up so you ensure no matter what happens that will get cleaned up right that was the idiom I I recommended to folks but the idea of with fixture is it's kind of like function compition that you can mix multiple traits that override with fixture together and then actually compose the with fixture functions so instead of actually invoking tests like that in the dark black what I recommend people do is say super with fixture test which is you call the super implementation of with fixture and pass the test function up and let it invoke it so if it's trait site then you know trait site will invoke it if it's some other other trait that has mixed overridden with fixture then that will probably call Super with fixture test and then that guy will invoke it so that's that was the idea is you can sort of compose these traits you can add different behaviors uh to do things all different kinds of things at the beginnings and end of tests or around the test function okay so that was 2.0 and that's what actually everybody's using these days unless you're on one of the milestones for three and then I also one of the problems with uh like the way junit did things with setup and tear down is that it requires a side effect and that's very unfunctional so what uh you can do like you can do uh set set up and tear down kind of thing in in a in a whiff fixture like this um so your setup fixture could create some value and stick it in a VAR and the clean up picture can clean it up if you want but what I what I did from from early on in 1.0 is is I had another kind of style another kind of Suite that was fixture Suite where you can declare a type of a fixture parameter that's passed into each test in that Suite so so so fixture Suite is a subtype of suite and fixture Suite has this abstract type fixture parameter and it has another with fixture method it's overloaded that has a different signature it takes a function one not a function zero and it's a fixture pram to outcome uh and it returns an outcome and then there is no implementation because I don't know what fixture pram is right I don't know how to create one and pass it in so these don't stack you would just make one at sort of the end point and the reason they don't stack is the fix prams will be can be different in each one right um technically you could if you actually have the same picture pram make it your own stack if you wanted but in general it's not going to be it's not going to stack okay so that's that's another thing and what um you know you might do in uh in your own test Suite is override with fixture in in in uh with fixture takes a test function that's fixture pram to outcome so what I've done here is I've fixed you know I've set the the abstract type to string and then uh I just create my fixture as the string hello and then I pass it in the test so when I invoke the test I pass the fixture into it and now all the tests get that hello string path to it so there's no side effect needed um but the same kind of thing happens you can do that in a try and then it finally do the cleanup um but uh yeah so that's that's basically how you use it okay so um an example of um I mean there's kind of two things you can do one is you can just do this in your test class itself and that's usually how people override with fixture it's just right in the test class but if you have if you find yourself implementing the same thing over and over you can pull it out into a trait and the way that would look is there's actually one in scol test called severed stack traces way back when uh idees did not actually know how to use the stack depth that's in the test failed exception in scol test and get rid of the the the scol test part of the stack Trace um if you mixed in this trade it would do that so it actually did kind of have a side effect but what it did is um exceptional basically in your your super WID fixture test asks the superclass implementation of WID fixture to invoke the test function and what comes back is the outcome and then you have a pattern match on whether it's exceptional or not so exceptional means it's either test failed or it's either failed or canceled because it has an exception in it and usually an exception is bad but uh in in a test it's actually good because you just found a bug right so I called it exceptional and so if it is either a test a failed or a canceled then um you grab the stack depth that's inside there and you sever it and so that's what um it just chops off the top part so that's an example of of uh you know overriding with fixture in a trait and so what what you say is that this severed stack traces trait extends Sweet Mix in which has a an abstract with fixture and then um then you have the self type of Suite so you're going to mix it into a suite that has actually implemented that thing um so what I do in the case of uh case of uh you know if it's not if it's either succeed or pending I just continue and return succeed and pending okay this sep it to your code as opposed to like the framework stack Trace so it like if you do an assert and that fails when you called AER it called into scholet test so that got onto the St that actually gets onto the stack and what scalet test is test failed exception has in it is a depth like you go three down that's your code the one that actually failed so what this does is it just chops off the top three which right because it just well it know there's a stack depth in the in the ex in the test failed exception so you you can uh ex you know For Better or Worse Java exception classes are mutable so you can mutate them uh even if actually I'm not sure if I can mutate the the stack Trace but if I can't what I do is I make a new no I can't I can't make a new one I mutate the stack Trace that's what you do and there's a way to do that in Java so I just mutate the stack trace on the way out so anyway uh what this is an example of is users will have can have these sweet mixin traits that you overhead with fixture and mix into a bunch of classes and you get the same with fixture sure in all of those classes okay so that uh existed and um what uh you know I I have to tell you that uh Monday I came back from Two and a Half Weeks in Asia so when I hit the slide that's when I remembered that so uh let me see what the heck I was trying to get at with this guy uh yeah this is just an example of of implementing um and I think it's another example it might be redundant this is an example of you can override uh you can extend a fixture suite and uh Define the fixture pram the string so users would do this usually in not in a mixin trait but in their own class that was what that's an example of and you don't usually uh call uh another with fixture that takes uh a fixture parameter but what you might want to do is call the width fixture the other width fixture up here that takes no parameters so there's two with fixtures down here in a fixture Suite there's the one you inherit from regular Suite that takes no parameters it's like no Arc test is what that's called and there's one down here that takes one parameter so either they're both and what if you override the one in here you may actually want to call the one uh with no parameters because someone may have mixed in for example separate stack traces and you'd want to have that behavior so what I ask people to do in the documentation is do it that way so instead of just saying with fixture and pass or test and pass in your fixture object you call the other with fixture method and you convert this test to a no Arc test and and to do that you have to give it the fixture so that's that's what uh there's just lots of code out there people doing these kinds of things that um that I want to keep working um the other thing that was kind of inconsistent in uh and I'm not sure how it happened I don't remember but I know that I made the type of the test body there's certain most of the Styles and scol test register tests at construction time as a function as a by name and in 2.x and in 1.x it was type unit so it's a bam to unit and uh I think I did that just because in junit was void and I said well that's unit so I made it that but over here in a fixture Suite you register fixture pram not to unit but to any and I don't remember why I did that I may have had a problem with unit down there but I thought that was weird that it was inconsistent or just not as nice so in 3.0 I wanted to make that consistent but I didn't know which direction to go does anybody have an opinion is that before I tell you what I ended up doing any any how many NES okay how many units no only one unit that oh no two units because it would have been a better joke if it was just one unit but anyway uh I thought so I actually agreed with you guys I made it unit and then uh I heard from users I'll I'll get to that later but uh it's actually any so I did go through a unit phase but it's any so that's um that's another thing that uh was you know in existence and this is the type of like see the the body in comment there that by name is unit in a regular fund Suite in a fixture fund Suite that is not a by name it's actually regular function fixture pram to any in 2.0 okay so to summarize 2.0 users can Define with fixture methods um the with fixture method in fixture Suites that take a fixture pram can delegate to the with fixture that takes no arguments and I recommend that uh in the documentation and and importantly according to the types the test is already completed when the function the test function returns so if you look at the the type of the test function it's it's either fixure PR to outcome or empty PRS outcome the result is outcome so the test has either succeeded failed canel or pending it's done right it's baked into the types and the test body result types are inconsistent 's unit and there's any right okay so uh I actually was teaching play like type safes play course two years ago or something and I was repeating the Mantra uh that uh there's three rules of reactive programming rule number one is never ever block you've probably heard typ safe people say that rule number two is never ever block and uh rule number three is well okay maybe it is not so terrible to block sometimes in your tests so that's a wishy-washy rule that they kind of try to like not put in red but it I was saying I was like repeating that because I'm supposed to in the class and then I got to wondering well why is that true why why do you have to why is it okay to block in your tests in play or in spray uh you can instead of blocking and finishing a response and then returning a response to the framework you can return a future response immediately so why can't you return a future assertion a scholet test and let it complete it right so that's what I thought so I asked the AA guys and they thought well you know maybe that's you know that sounds interesting whatever and um and I kept thinking about it and I thought well that's interesting and I thought of a way to do it and on a on a train ride I just coated it up and and kind of got a proof of concept that it was possible but I just it's one of those like that voice in my head was like if you add this are you really solving a problem that adds complexity that isn't really needed right it's that thing of I don't want to screw it up so I really wasn't going to it and every time I'd go to conference I'd ask people what do you think of that and they they'd all kind of say yeah that seems like you know but there was no burning need right so I I know I wasn't going to do it uh yet but um it turns out that there is one simple fact of jvas script is you can't block and I actually didn't understand this right away when we did our port to scholar JS I thought I'd gotten to the end of the tunnel and then I got this this issue posted uh from a guy whose name is code um and he just showed this simple code here that in our you know 3.0 Milestone uh didn't work and the what the way JavaScript I'm not sure how many of you have had the pleasure of using JavaScript but there's just one thread going around in circles there's an event cue and the thread just picks up a job and does it and the same thread picks up a job and does it picks up another job and does it it's kind of like the awt event thread or the swing event thread um that's how it works so it's single threaded so can't do anything multi-threaded right so you know that's that's what I thought so all of our tests pass and we actually had scholar JS working schol test working on scholar JS last summer and I was about to release rc1 thinking hey I'm I'm almost there and what this guy did is he actually had the audacity to call into an external an API so what you can do from JavaScript is call into an API that is now uh outside of the JavaScript VM or whatever you call it it can be a c thread a different thread that does something and when it finishes like maybe it goes out on the net and does a request and when something comes back it sticks a job on the javascripts of NQ and then eventually that that one JavaScript bed is going to pick it up right so what we had for futures and they did Port Futures to schola JS it's pretty interesting um was this this Scola Futures trait that has blocking things so like uh here it is called future value x. future value future value blocks until the future named X is finished and then it returns a value and if it doesn't complete in a certain amount of time it blow you know with your patients configuration it blows up with test failed exception or if it or if the fature fails it also throws that failure so that's what feuture value does um so on schol J it didn't wait because it can't you can't block on in JavaScript so it basically did this assertion too soon the actually really acnc thing hadn't come back and when I saw saw that I'm like oh no I have to do that async testing thing where you can return a future to the test frame where the only way to do it because then he could actually put he could on the X future he could map an assertion and schol test would just hold on to it just like when play is given back a future response it just holds on to it when that future completes it sends the response what scholet test would do was when that the future completes it performs the assertion and it then it then at that time it re does event to the reporter saying test succeeded or test failed right so that's actually what's taken another six months was async testing to get that to work um so the first problem is future of what so I say future of assertion but the type of assert in 2.0 was unit just like you would expect it isn't it's always been unit right so I didn't like that and so I I really wanted I wanted to be able to EXP Blain this by saying future and then in square braet have something called assertion so what I did is I made a type Alias assertion that's the one and only one succeeded Singleton so now when you type uh assert x equals 1 and it's it's valid you get a you actually get a succeeded Singleton back instead of the unit Singleton right and it actually looks pretty but now uh and the same thing so we went through and did that for um uh both assertions and matchers right they all return this they have a result type assertion with and it Returns the six seeded Singleton or it throws an exception because it it they would break code if I made it not throw an exception so it still throws an ex an exception same exception as before if it fails but um instead of returning the unit value Singleton it now Returns the succeeded Singleton okay um so um what you can then do is we made um we we couldn't make a trait that mixes into funu to make it async because funu has a WID fixture that assumes the test is finished when it completes right so that wouldn't work so one thing I could do is break all that which I can't but I could and redefine it um but what I did is I made a different style tra called async Fun Suite async prop speec async you know word speec where the type of the test body is future assertion so if I have a future then I just map onto that value and assert and that is now a future the type of that is future assertion and that's what it's being looked for by the test body so that will compile and uh it just you know behind the scenes when that future completes it'll fire the events to the reporter okay so that worked uh that actually solved the problem that goty you know it fixed that guy's problem uh so instead of using Scala Futures you just use an async testing style and um these guys as I mentioned won't work because they you know the test result type is outcome and I really need a future outcome or something like that right so um I really struggled with how to do this um because originally I just in async test Suite I added an overloaded one and made the original one final uh but there was uh again I was about thinking I was finished and some hio asked hio seberger said hey can you how can you limit the time that you wait for a future to complete once I give it back to you I'm like uh I don't know so uh when I finally went through that process I realized I I need to do one small break uh which is I made two subtypes of site I took with fixture out of site and I moved it down to two new traits called test suite and async test suite and what test Suite does is it it adds the type of a test to it so the type of a test in test Suite is empty PRS to outcome the type of a test in async test Suite is empty prins the future outcome so that's and so there's two different width fixtures down there um so originally it was future square brackets outcome but then when you it was actually quite clunky to transform it in the with fixure method you've got to actually transform the future and so if you want to handle if the test failed that's actually a future success so you have to handle success failed and I thought that would be very confusing for users so what I actually made while I was in my Asia trip is I made something called future out where the methods are on succeeded then on failed then so they actually fit the domain of testing and the other thing they do is that's like a side effect right if you want to clean something up um it was actually quite clunky in the future API to handle whether something succeeded or failed to give back a new future that uh didn't complete until the original feature completed and your your call back function completed um you have to say map and recover both or recover first then map right recover handles the failed case and map handles the other one so all these methods in future outcome return a new future outcome that will not complete until everything happens and what I'm trying to do is make the semantics of working in the future space space the same as working in the regular space and so in the regular space um if the uh you know if your cleanup method like your cleanup code and your finding Clause blew up with an exception actually your test failed with that exception right in the Futures API there's a method called and then where you can register a call back but if it blows up it just it just hides it right that's a it's not necessarily wrong but it's not consistent with the other one so this guy if you know anything you do like your call back function blows up with an exception now the test fails with that instead just like it would over there so yes can I use only ask and F I want to so why do I need if I'm moving forward can I use only test uh you can use only async test if you want yeah um but um if you call get you need to come up with a future somehow to oh no you don't there's also an implicit conversion from assertion to Future assertion so if you just want to have one that's synchronous mixed in uh you can in other words in an async testing style you can have both tests that return future assertion and tests that return assertion mixed together that's okay moving forward t no there's millions of lines of code out there that would be upset if I did that so that's that's one of the thing like even starting from scratch let's say I was starting from from scratch I could do just the async one because it's more General because a synchronous test is a special case of the async where the future is already completed right as you pointed out um but what that would mean is that even when you no matter what kind of with fixture you're using you'd always have to be transforming futures for beginners I think that's a little bit of a steep hill to climb but it's kind of moot because there's just too much existing code out there um so I can't so I I that's and this is the the fight I've been fighting is like trying to not let it explode um another another example that I I think about is J you know atlassian has anybody ever yeah that has got so many features it can it's so powerful I can't figure out how to do anything simple I mean I only want to do like a handful of things and I don't want that to happen to this and it can happen right I could screw it up so like here now I've exploded my testing cells to two like double as double right so anyway um but uh that is a breaking change because with fixture used to be up in suite and now it's in test Suite but all you have to do it won't break any any with fixtures you overrode in your test class itself it'll only break these mix in traits which there're aren't that many of them I don't think and to fix it you just add the word test where there where it's dark so it's just a textual change it's not too bad so I think that's going to happen in 3.0 um but this hasn't happened yet in a milestone release I just kind of came up with that name a few weeks ago um okay so the other thing that didn't really exist nicely on future was a way to do this try finally so I made a a a a try finally for futures called clean up lastly no no complete lastly and this is not um fixed with font but if if it when it is I mean most times people program with that they actually line up those curly bles line up so it looks kind of pretty but complete is like try it says complete this futuristic thing this is can be any type te with a with a futuristic of te type class it works for future of any type and it works for future outcome that I invented right um so it says if if the body in between the complete curly braces blows up with an exception it will definitely do the cleanup so if it blows up if that blows up otherwise it's going to return some futuristic thing so it registers the cleanup code to happen after that future completes and returns a new future or futuristic thing it'll be return a new future outcome in that case uh that doesn't complete until both the original future completes and the cleanup is finished and if the cleanup blows up with an exception that future will have it as the failure right so so that's now the recommended way to do um Cleanup in the future in the async stells in the current Milestone release it was called with cleanup and uh someone told me it looked like a bug so in Japan so I I I just changed it to that recently too I think that's uh probably what it's going to be um and this is basically what I ended up with and again it it used to be much simpler there used to be site fixture Suite now there's four other things there that in the surface area um the width fixtures are defined in test suite and async test suite and then there's that fixture Suite that that defines this abstract fixture param type and then now there's fixture test suite and fixture async test Suite uh again that that uh add the WID fixture so you'll either go fixture parameter outcome or fixture parameter future outcome so that's that is what I ended up with uh and and the the last time I I gave a talk was in Scala Matsuri in Japan about 3 weeks ago and uh I hadn't gotten I really you just still didn't like this cuz it was too much stuff I hadn't come up with the names test Suite ning test Suite yet but when I when I saw it on the screen I was like well looks kind of like a rocket so maybe it's okay um and I think the name test Suite actually works pretty well because I've never used that for anything and it's it's an abstraction of Suite where the test function is defined so I think that kind of actually makes sense and is explainable so um another thing that in skull test 1.0 uh wasn't really going to work I mean I had for a long time think been thinking there should maybe be a type at the end of a test that's not any or unit something more strict um and so if there's going to someday be a assertion type at the end of a test then uh intercept would not work at the end of a test because intercept is this thing that will make sure that this code between the curly races throws an exception and if it does it returns it to you so you can perform more assertions but now that has Type well this one has Type string index out of bounds exception and I had this other Syntax for Mattress evaluating this bit of code should produce a string index out of bounds exception that did the same thing right so this one says had because I actually deprecated that in 2.0 already kind of for this reason of I I needed to separate checking for an exception that I don't plan to ever look at again so I don't need it returned and checking for exception that I also want to do further assertions on um so I deprecated that in 2.o and I actually that's removed in 3.0 that's been deprecated over 2 years um so now what you have in in 2.o um which is the new way to do it is you can first of all if you just want an exception you can say the exing in X Out of Bounds exception thrown by this code and you get it um you can then perform an assertion on it like X should have message blah um that at the end of a test would work because that's an assertion right um or if you want to do it together you can say the string out index out of bounds exception thrown by this bit of code should have message blah that's an assertion because there's a should there um or you can just say if you don't care you can say a string index out of bounds exception should be thrown by this bit of code and you don't it doesn't return the exception that one because you didn't say you wanted it right that one has result type assertion so that can end a test right so in 3.0 I added one more intercept like thing called assert throws that does the same thing is that last thrown by it just checks to see if there's an exception and it doesn't return it so just assert that this code throws this stuff and I'm done right so now that can end a test but there are gazillions of tests out there that end an intercept that would break if I just Chang the result type from any to assertion and that would really make people mad um also these things don't work in um they don't work in Futures because in Futures there's this difference between failure and success right so what was needed was a way to say I expect this future to fail with this exception so I had to create these guys and it's the same kind of thing the first one recover to exception if uh illegal State exception if that code that code in there creates a future right uh empty actor question mark Peak will give you back a future of something if it fails I expect it to fail with with illegal State exception um if that doesn't then I will get uh this will transform to a future Asser uh what will this do this will transform to a failed future of whatever I'm not sure what type actually but it's a failed future that's where that goes um if it succeeds then I actually change it to a future exception so now you have an exception in the future and which you can do more assertions you can map assertions onto that exception right um um R to succeeded if this future fails with a legal State conception if that happens it just gives you a future succeeded right so that that's a future assertion that can end the test this one has result type future of illegal State exception this one has a result type future assertion so it's kind of like intercept and assert throws for future space right so that's defined in async Suite in 3.0 um so I had to invent that and then um there's another problem with having a result type of assertion which is uh there's something called I call I called fixure context object was context is a terrible name for anything but I couldn't come up with a better one uh and what this is is if is this kind of uh you can make a trait that has something in it and then your test body can actually just instantiate that and now you inherit that fixture sort of the fixture is the context your test is in right and um um that is pretty code and a lot of people wrote code like that because I put it in my de documentation except now the type of the body is buffer it's not assertion right um that's that's one problem and another problem is the uh there's a lot of external libraries that do different weird things right so I can change the assertions in skullet test a return type assertion but the uh Oates has a bunch of a bunch of things that return unit or the message type so expect message actually Returns the message it's like Intercept in AA test kit so that would not work at the end of a test and there's just tons of test code out there where that's at the end right um okay so those are some of the the the issues that I was worried about so I did add I mean there was there's a fail a cancel and a pending in 2.0 and in fact in 1.0 those three EX Ed so I added succeed and succeed just is type assertion and it returns a succeeded Singleton so you can just stick that in the the test to solve any of these type eras but it's still it's actually probably not too painful to do that as you go but if you have a a huge test suite and you have to go in and retrofit it would be very frustrating for users I think um and what I did though is I I actually recently created a another thing called assertion which is or scal test compatible assertion and currently it's written in Scala because I couldn't get it to do something but the idea is that that would actually be a Java interface and it would be released in a separate jar it's just an empty marker interface so there'll just be one doto of that and there'll never be any other release because it's just an empty marker interface and oh unless I have to worry about binary compatibility of java releases or something but uh my hope is I'd never have to release another one right so AA test could could actually depend on just that jar scull test compat ass cert. jar it has one Java interface in it and they can actually have their you know their expect no message return some instance of that and that's what the result type of tests are in scholet tests so that it would be more convenient for for external libraries like scal Mo its assertions could actually just work at the end of a test right um and then I also made something called fixture context that's actually a trait there never was one I just used like the words fixture context before now there's a fixture context trait that extends assertion also so I can just make trait buffer extends fixture context and now it is also a compatible assertion and can actually this this syntax will now work right be readable in new buffer new buffer makes an instance of buffer which is a fixture context which is a compatible assertion so it will actually work um so so I did that and uh that was recent and then what happened with this is um this is currently what we have uh I did change those to any and the reason is is when they were unit so like I said I first changed them to unit so they're all unit they're expecting unit and uh Scala is happy to discard any value to unit so I thought it would be fine but um and I don't care if it's discarded because I don't care what it is as long as it isn't an exception the test passes right but what there's a compiler flag in Scala that says warn on value discard - y warn on value discard and if if you like have a three um and you pass it to something that requires unit it will just stick a semicolon unit there the compiler will do that but it will actually warn you so suddenly everybody got a warnings because the result typee people who had that flag enabled the result type that was expected was unit but an assert at the end was now returning succeeded it used to return unit now it didn't right so that's and then there's another flag you can set saying any kind of warning make it not compile so it's like killed everybody's compile yes I've also had tests where I've had to throw a unit at the end of it it's like the compil like oh what type is it supposed to be huh and then I'll have to throw for whatever reason conf the type and I have to throw just at the end of my test for whatever reason I wrote it with kind of work obviously going to throw yeah that would happen if if uh if you were like down here where there's a unit if it's units required and you have that flag set and you have like the flag that not compile set then it will actually not compile unless you stick a unit there and that's what people are having to do is sticking unit everywhere and so that's that's why it's any any was the right answer eventually so I switched it to any but this is still kind kind of like inconsistent what I sort of feel like if I had to start over is that it should be assertion fixure pram assertion empty PRS assertion future assertion right it just to be assertion everywhere and I think it wouldn't be that cumbersome if IID started that way yes can I ask you sure I'm going to find my the first slide I that you showed yeah was that you did not like a string could be equal to an right implying why are we subtyping but a lot of your other slides are likeing a lot of mixing a lot of subtyping do you use this much mixing and subtyping in your actual runtime production code or is this like something you in test frame uh sub typing is not evil in Scala it exists obviously yeah but the the actual the problem with the the equality comparison is just that the there's it's type any basically uh Java's object Java Lang object has an equals method that takes an object so we we didn't have to just integrate with it the way that we did uh but we did and it it's actually makes it kind of seamless to use all these Java things but then you get that typ ail obviously well it's not necessary but it it it's like if you know if if Martin had gone different direction then the harder he makes makes it for Java people 5 years ago maybe people wouldn't have adopted as much I that missing implicit like we could have we could have had class we want to yes and you still can but uh anyway that decision was made um I think that tra basically what I use in scholar test is traits traits have are actually the way traits work I think makes o work where you can actually do this you can't do this in Java it's really really clunky but because you can you can actually have a uh abstract override you know that way of mixing traits and overriding the super call super even though it's not defined is what makes all this work and it's really it's just a kind of function composition you mix in mix in mix in and you're like functions are calling each other so it's kind of an objectoriented way to do function composition so that it seems to work pretty well in scholar test um anyway that's what we have is that yes question sure so theoretically actually should run much faster you're not working on the if you have multiple weights it's actually going to finish as the longest wait can question yes um that was he said he his theory in theory that the async may run faster because it's not blocking because if you have multiple weights right if you have multiple weights they're serialized so possibly I didn't believe it I would never believe that until I tried it and I've never I've never tried it I didn't do it because of performance I did it because of scholar JS and again even going back to that original point I hope someday I don't look back and think it was a mistake to do the scholar JS support right because I've had it's sort of taken me in a bunch of different directions that add a lot of surface area but anyway um there's a problem with running faster is that what happens faster for sure is the test body completes faster which returns an assertion so you could very quickly get a million Futures trying to compete for each other and then they'll all time out that's the problem I did worry about I that was my premature optimizations I was worried about that so the default execution context I made a new one called serial execution context where when you do your mapping and you know when you like transform your future there's always an execution context implicitly pulled in that's used the one that's there by default just remembers what you asked it to do like puts a job in my own queue and then when the test function returns before we move on to the next test we actually go through and have the same thread execute them and the other thing that does is you don't have to worry by default about synchronizing axis from multiple threads so if you if you really want multi-threaded in your test then you just override the execution context and and put in global or whatever you want but if so if you're using default if the Q empties but the test hasn't finished I block on the jvm because otherwise I could end up that's how I actually throttle them so if you can basically say if I have eight threads I may I can just have eight tests running concurrently to so that they each have enough threads available to complete or you could you could throttle it to 16 or whatever but uh if you just override all your execution context to Global they'll just all start and then some of them may time out but you can tune it right which mean I can't find appc the default ex correct but I actually got the feedback that that wasn't what they wanted um finding bugs by just hammering threads at it you can do uh basically if you want to do something multi-threaded currently you you have to actually select a different execution context and that also kind of fit with the way I did parallel test execution in the past it I make a new instance so that you don't have to worry about synchronizing different test access to instance variables because people do that they make vars because they're used to junit so if you run them in parallel you mix in parallel test execution each one each test runs its own instance so they can't really clobber each other so you don't have to think about it so that's what I I did before and I kind of that's how I kind of went this direction as the default it's not set into stone though and in JavaScript I in the current Mouse release I have the Q as the default using the JavaScript queue but that doesn't actually guarantee the order so we just changed that to this so if you actually want to do something truly async currently in in like 3.0x branch in JavaScript you have to if you have to do anything multi-threaded on either of them you actually have to set a different execution context that's basically I think that might make sense because it's consistent so but that one isn't isn't really decided but uh anyway let me I I should move on because I want to let uh our other speakers have an opportunity here um so I did try actually implicitly converting any dissertion to S to like have funset take type assertion but old code work and one problem I have with that is it's an Abomination I don't want to convert any to anything even though I do control one type it just seems kind of wrong philosophical to me it's too too clever or whatever it seems too prone to confusion but it also doesn't work because remember I said there's this expect message method in aaes kit it infers Type M by looking at the message type and then it returns it right so you say expect the string hello um it's going to infer M to be string and the result type of expect message is string so this implicit inversion what you wanted to do is convert the result of expect message right to assertion it basically discards that string and puts a succeeded but what actually it does is it converts the the parameter you've passed to expect message and the test fails with uh expected succeeded but got hello right so that just that would break code and so I I uh I mean for a while I actually had that in there and I just it was wrong so I took that out um and one of the things that isn't going to be in 3.o but probably later will happen is this thing called an expectation which is like an assertion that returns a result and doesn't throw an exception and I want to put that in there probably for laws testing and again it's you know I may not ever do it because it might fall into that thing if it's not needed enough but that's one of the things I'm thinking about is is I think this might be nice to have and uh it actually works so we did flush this out as well just to make sure that it worked like we thought it would but then I I got rid of it because I don't want it in 3.0 I don't want to spend time uh getting it working and completed and documented and everything but if something ends in a fact which is the the typ alas expectation is equal to a fact and a fact is either a yes or a no so it's it's kind of like Boolean logic but with error messages not just true and false it's yes something happened right it's an observation of of an experiment that you perform formed um and it's more than just Boolean logic though because it also has an uh is equivalent to and an imply so it's really like predicate logic or proposition logic especially if you add existent for all in there which you can use property based testing for so you can really make a test as a statement of logic and it will be a value of type expectation and then I I can have I'm okay with an implic conversion from expectation to assertion and that's how that would work because when people when a test fails they actually want a stack Trace so when you convert an expectation to assertion if it's a yes it just gives you succeeded if it's a no it throws test failed exception with an airor message and a stack depth and all that stuff so um so anyway that's kind of one of the use cases for having assertion be the result type so this idea I just had you know I really struggled with how to do it and I I have had an idea that I've never told anybody because I had it a few days ago these are all the style traits and scallet tests that you register tests and so they have type any right now um you know perhaps your favorites are among these maybe not I don't know but I have the thought that uh you could actually deprecate all of them deprecate fun suite and that's absolutely crazy but I think I mean if I'm ever going to have a style where the type is assertion it should be called Fun Suite it should be like the the name for it so what I thought I could do is just have this you know it's more bloat but I could ask people basically when you use fun Suite it still works exactly as before but you get this deprecation warning that you need to change the name to any fun Suite so it's again it's one of these textual things but people have to go all over the place and and maybe they could write a script to do it but it's it's painful to put people through but then you fun suite and any fun suite are the same thing the old name gives you the deprecation warning down the road I could change fun Suite to have type assertion and it would be you know achievable I think um but now I've again I've got bloat and that's in it's in an experiment Branch I didn't even put that in 3.0x so anyway I would enjoy hearing you guys didn't run screaming so maybe it's not too bad but uh I just there's just so much cat out there I hate to make people go through that if I'm not uh you know I just hate to make people go through that kind of thing even though it's a deprecation cycle it's still pain so anyway that's my whole thing there there there I did actually touch on serial execution context that's the default so I just you know thought for Q&A here's some ideas there's that thing if you want to talk about that tests actually execute one after another that was a request I thought they should run in parallel by default but even though they're async a scholar test won't start the second test until the first test its future has completed and the reason they did that is basically people are reusing fixtures like a database and they they don't want them running in parallel and in parallel test execution Works before and after work so everything is async and I thought it was async in 2.0 um because the after all method in 2.0 I made sure it doesn't com doesn't execute until all these tests finished in parallel in wo that wasn't true but there were bugs in there that weren't hit by people um and then just were features that didn't exist so we had to do a lot work to get all that stuff to work but basically your async Styles you just mix in the same traits and it works the same essentially before and after work it's all async um parallel Tex extion even works on scholar JS because if you have a bunch of tests let's say you have 10 tests in one suite and each one of those calls into the one of those API calls that uses a different thread to call out into the net and that takes 30 seconds it's kind of nice to actually get 10 of them going at once get those 10 AC going at once on schol on JavaScript and so your tests are actually running a parallel so it even makes sense on JavaScript so anyway those are some ideas but I just uh appreciate being invited uh by Alexi wherever you went and uh you know would welcome any kind of feedback because none of this is actually released final yet so I can still change it thank you let's do maybe one or two questions so get speak but I'm so uh does Happ to me with some of my tests love scell check you have smell Che integration obviously it's one only Integrations but let's say I'm doing something with like a mock framework where I'm expecting mation or to have like a protocol y I'm expecting certain calls to happen but iteration of the scholet of the scholet check doesn't have its own like cut off now I don't know if this is like probably all this comes down from junit what it means for the end of a test yeah is this something you'd have to like recreate what what it means to have tests to have a cut off and like did your did your calls get made in this one out of 100 Loop or is that something that you can address uh I'm not sure we might have a cut off so there's timeouts so that was well it's not a timeout so let's say I'm saying all right I I I do this call I expect because I said f obey I expect my unit to call some other M called like bar with with a correct so I only will find out if this didn't happen if I'm doing a skull skullet check at the end I mean if it calls the wrong order yeah but only at the end of my with I realized oh it didn't actually call any of that okay yes so signal I'm at the end of a loop I'm at the end of one property um is that something that nobody's brought up uh yes nobody's brought that up it's amazing actually one of the things I find kind of surprising is people just I guess maybe because you don't pay for it no one complains until until I happen to be there then they bring it up and I'm like well that I was good to know like like that thing where people were having to put the unit everywhere I me these good friends of mine and they were just just doing it but then when when they I saw them at a conference says hey by the way I was like oh I didn't know that uh so I'm sure there's been people who've had that pain but they just they just feel deal with it but afterwards maybe you can talk to me about it yeah is there one more don't let him go going once [Music]