scala.bythebay.io: Leif Wickland, Toward a Safer Scala
Recording: scala.bythebay.io: Leif Wickland, Toward a Safer Scala
you hi I'm Lee frequent I have been doing Scala since about 2010 at the time I was working for a company called the rubicon project and I got to go off and build a system to count usage data for financial reasons that company got bought by oracle I stuck around did my time I can promise you they're still Scala in production at Oracle because they need to count the money still I moved to a company called rubicon project I ended up doing something very similar there I have the pleasure of doing all this from Bozeman Montana which is the lovely lovely place this isn't Bozeman but you know it's similar so what is static analysis I think it's writing running software on your code to detect problems before you run it to me that sounds a little bit like what I want my strongly typed language to do for me in the first place but really this isn't quite the compilers job so let's go on a little bit with that so those compilers job really is to turn the code that you write into something that executes and along the way they detect some conditions that probably aren't optimal and you don't really want to ship and they can report those to you but by default they usually don't I think we should have coding standards I think coding standards that are a dead document just sitting there aren't really coding standards and I think if you want to have a real coding standard you need to have that automatically unforced I think code reviews great but I think we can automate a lot of that if we use static analysis well so what you're going to get out of static analysis what you're going to deserve what I think we can deliver you is a better compiler that causes you fewer errors and explains things better when you do mistakes I think you can have more hygienic code that's more consistent that's easier for your team to read and doesn't allow itself to devolve into worse bad things of like infinitely long functions and gigantic classes I think we can have higher quality code that doesn't let people coming into Scala fall back on the bad practices they learn from before you know null and VAR here and I think we can ship fewer bugs so I'll show you some examples of those coming up so there are lots of example lots of tools that we can use for static analysis in the Scala ecosystem I'm not going to give you all the details on all of these i'm going to say some of these are good and some of these are better but i want to point out before i start casting aspersions in any way that i'm really thankful that the ecosystem has all these tools and you know express some gratitude before i started ripping things down so moving on your ide probably has some ability to tell you that you're doing things wrong so for example if you are editing things in IntelliJ it'll highlight things with a dark yellow color and that's usually tell you that like there's something you're doing there that's suboptimal I'm not going to talk about those very much because I don't really think that they are the tool that you need because it's hard to apply that uniformly now when i was at oracle there were some teams that mandated everyone on the team used the exact same ide and they would do their static analysis that way which i found sort of repugnant and repressive and i hope you don't do that and i also hope you don't build your releases out of your IDE so I think that your tools for static analysis need to be independent of your IDE I mean they can supplement in the IDE but we need something outside of there there are some web UI tools that will do static analysis and those are kind of nice if you have a mixed set of languages that you'd use it can need a way to like provide your executives some view of code quality across Python and Scala and all kinds of things these can be helpful the analysis that the code asset II and scrutinize er have for Scala at least last time I looked was somewhat rudimentary and they didn't have sort of that full suite that I want to do so I'm not going to cover those particularly either but they do exist so for the things I am going to cover let's approach them with these three criteria how high how much reward do i get for using them is it giving me lots of false positives that waste my time and make my team angry or does it actually helped me find errors that matter suppression when it does find something and it's not actually a thing I care about or I really really do need know for some reason is there a way that i can say on this one line let me have my stinking null I'm sorry and how quickly can I get started can I get it integrated in my build quickly and can people who see these errors easily move on figure out what's wrong with them and and keep going so the here is an example of some Scala that doesn't quite do what we want can anybody tell me what this is going to do it's going to print zero right this is not what we want anyway because our friendly valet there gets initialized to be which has a private backing variable that got initialized vited JDM 20 and then eventually that thing got set to 42 later on so this is not remotely what we want and I don't want my compiler to let me do that that's the reason I'm using skull and not JavaScript right i mean like come on let's do good things here so the good news is that scala 211 now warns us when we do something dumb like this the bad news is that it's only a warning and I can still ship it and I don't want that to happen I want to find out a compile-time not run time that I've done something stupid so Seth recommends that if you've got a well-managed project that you set X fatal warnings do you have a well-managed project I mean show of hands like who has X fatal warnings on ok not enough right like this is a really useful tool that this compiler will give us and this isn't the only thing that you'll benefit from there so in SBT you do something like this to turn on those warnings in a way that it makes them not warnings makes the mayor's what is this code do this code is always false it compiles we pass unit 2 equals equals says a set of 1 is not equal to unit right this is not what I want to have happen I want my compiler to tell me I'm wrong what does this code do likewise this code passes unit to set to the apply method on set and says false no there is not a unit member in that set again I don't want this to happen Scala deprecated this feature called adapted arguments around 210 but it's just deprecated you can still ship this you can still get baffled as all can be by these this code the compiles and doesn't do what you want so there's a switch dash deprecation that will provide useful error messages and when coupled with ex fatal warnings turns this bad stuff into an actual compiler warning that stops and lets you know that you've made a mistake one problem with turning on dash deprecation is it applies also to libraries so if you go and you know upgrade to a new version of a library and they've deprecated some functionality that will appear as a compiler warning now maybe that is often what I want because I want to find out that I really need to stop using that method but it can also be a problem if you're in some horrible fire and you need to upgrade a library to work around a bug and yeah just something to be aware of what is this code do yes and when we run it what happens yes exactly this code blows up when we ship it it compiled just fine but because the format method doesn't receive as many arguments as the format specifier has placeholders at runtime it blows up and this is probably the kind of thing you do in an error message so the fact that you had an error and now you've masked that error by producing a new error is awesome right just perfect I desperately want this never to happen again so the good news is Scala gave us string interpolation right I love string interpolation because it makes it's so much harder for me to make that stupid mistake but the thing that happens to me is is this now I ship this code and my logs are now full of strings with dollar signs in them because I'm an idiot and I forgot to put an s on the left-hand side of those strings so I didn't get string interpolations I just got lots of money and while I love money I don't want the money in my logs so good news Scala will tell you when you do this stupid thing if you ask it to it will report a missing interpolator exception in this case to get that you need to add a dash X lint to your Scala see options and then you'll have these lovely lovely warnings so excellent is kind of a big hammer it enables a whole bunch of things at once as of 211 for you can get one at a time you can you know incrementally enable them which is really nice if you're trying to add this to an existing project that probably has some unfortunate mistakes what does this code do does it compile I mean not in my project it doesn't but sadly vanilla this project this will compile and i'll stick a integer in that sequence with those ADT members which is not remotely what I intended to do the compiler infers a sequence of any and you know this this isn't probably what I want to do if it is I'd be happy to tell it yeah I really mean that so by here's another way we can look at this same kind of problem I can take the cross product of a sequence of integers and the sequence of strings and get out something nonsensical that is a mixture of integers and strings right and it will infer it as any which is good for approximately a universal equality and universal to string both of which make you know certain faction of our camp very angry so if I want to prevent this from happening I can enable X went and Scala will give me this glorious error message that says any was inferred and the world is a much better place all of a sudden so back to this go home add those things to your project you will find mistakes most likely and if not you won't make as many in the future so on my delightful scale I'd say that the warnings thats call us he gives you a really useful there's no way to suppress these things so I hope that you know that that's just the way it is and their documentation on excellent and so forth isn't fantastic by any stretch about the only documentation is what you get off the command line by passing dash help Scala see and yeah I don't have a copy of Scala see sitting around because I just let s BTW download it for me so it's not easy to get to necessarily one interesting quirk hear that Stephen compile pointed out to me is that if you are a library author and you cross build for ask Allah to 10 and 211 and 212 to help the community out this can be a problem because the options for static analysis to scala see are not consistent across all those versions well the solution that somebody pointed out i can't remember now is that you just configure SBT only apply those switches to the most recent version of scala and don't worry about it on the other one alright other static analysis options in the jvm ecosystem there's a thing called fine bugs it's an old tool that came out of the University of Maryland a guy made a plugin for SBT it used to be that if you want googling for static analysis for scala this is one of the things that would turn up don't do it it's terrible fine bugs is looking at byte codes and it sees the things that scala see generates and just freaks out dumps lots of error messages and it is totally useless so the signal-to-noise ratio is a real bad don't do it just walk away other options over in the static analysis world there's a tool called Scala style as the name implies it's kind of towards style so it helps you go down that path of an executable coding style in a lot of cases it's really easy to get started with it's got a nice SBT plug in the Coursera class that teaches Scala stuff uses Scala style to add that extra layer on top of helping people right scala in a functional way instead of scala as a better java and it's a mature well-documented project the bad news is that configurate you end up writing a bunch of XML if you're still with me the good news is that we can incrementally change things so it's easy to just have a config file where everything is disabled add it to your existing project add one rule enabled and then you know periodically you know tighten the screws down a little bit it's got nice suppression mechanisms where I can bracket a region with comments turning things off and back on again it's got a more concise form that you can use to do things like this as well if I make broad sweeping strokes about the rule set that Scala see offers I'll go with these three so like I say executable coding standard i can put caps on how long lines and files and functions can be I can keep sick lomatic complexity down I can limit the number of arguments to a function I can prevent magic numbers from appearing in my code you know like if I have a constant integer I must assign it to a vowel directly instead of just having it appear in the middle of an expression so that's nice stuff I can ban people from using null VAR return while and calling clone and I can encourage good practices like don't ship code that has public methods that aren't type annotated right doing that shipping public methods that aren't annotated is good way to break your users later when you make a change inside and Scala see infers a slightly different type so there's a lot of wisdom baked in here that you know this tool automatically gives you so if you want to extend Scala style to do other things for you there are two different routes you can go you can just write a regular expression to match code and if that regular expression is found in your code it freaks out so one of those that i use as i say i don't want to see print line in my code when it goes to continuous integration and you know i can suppress that in the few places that i need it but in general if we're wandering around adding print line to our code to like quickly figure out what's wrong I don't want that to accidentally then get shipped so this helps me if I need a more complex custom rule they have a way to do that the API isn't awesome and the documentation isn't terrific but it's possible so the rule sets sort of broad they'll cover both sides of a controversy so if you want to have spaces around your operators they can require that you do that if you want to ensure they're not spaces around your operators they can also ensure you don't do that so like yeah it's a big bag of tools so in general I'm pretty happy with Scala style next to I want to sorry so I think Scala style is useful because if you're coding standard isn't automatically in uniformly applied you don't have a coding standard I think what you have is a vehicle for misanthrope e so when I've seen coding standards before their big long dead documents of bullet points uh bullet points and sub sub bullet points that only get referred to when somebody's angry at somebody else and they want to hit them over the head with it to make them do what they want right that's not what I want I want just a set of rules everybody does agrees to play by so I think Scala style is useful for that on the topic I really like school era more it's a tool that automatically formats your code I don't think you're a snowflake I don't think I'm a snowflake and I don't think my indentation style is amazingly better than yours I just want us to agree that we're going to have consistent formatting and we'll go with it so I've been happily using scelera form for a whole bunch of years and everything ends up looking about the same because of that there's a nice house BTW plug-in that you can use to either compile every time code changes or configure it differently to only run when you tell it to there's also a new thing like this that I like a little bit better but I haven't used in production yet if you want to describe it that way called a scala format it's got a lot more advanced options so one of the things that scala style doesn't do is automatically wrap your code skoloff sorry scalera form doesn't do that scala format does automatically wrap your code it has a lot of neat and alignment options I'm a big fan of it it also has an interesting option to run a command that just tells you whether your code is formatted correctly which is useful for CI so yeah I I would definitely look at this as an alternative to Scully reform if you were thinking about going this direction okay another static analysis tool that exists is abide it started as a summer student project in 2014 it's owned by the light then typesafe folks the bad news is it's never really been released it's not a project that's reached critical mass but it has a tremendous amount of potential in the sense that it has a really good architecture for analyzing code and it can do so more quickly than some of these other tools that I'm talking about because it's built to make the analysis happen in fewer passes over the code but you can't really recommend it because it's never been released so it's got some interesting things like unused local variable detection is a feature i really wish i had and i don't think any of the other tools have that and it would make sense for the nice folks that type safelite done whatever we call them these days to have this because it means that you get to take some of that complexity out of scala see that they're doing today and move it off into another tool so that they can innovate in Scala see faster but again there's not a community around this so Adrian would be really happy if somebody would come along and help him build this thing up into something better I can't I can't recommend it can't really evaluate it doesn't exist but you know it's close to existing sort of all right what about this code was this code do to us sadly this code compiles and we get a sequence of product with serializable which is one of my least favorite things to have in the world because the top one is a 2-tuple and the bottom one is a 3-2 pool and through the magical power of inheritance those are the same kind of thing and our type inference system says yes I can do that for you well oftentimes if that happens it's not what I want Wirt remover is a static analysis tool Brian McKenna wrote it is accurate and well-documented it has a nice SPT integration it will detect that problem that we just talked about you can extend it neat ways and it has really good suppression options via an annotation for a small area or just exclude a file from analysis entirely if I broadly categorize its rules we've got these three so problematic inheritance like we just saw if it ever sees the compiler inferring these things it will complain banning better Java just like in Scala style kind of the null VAR return and throw or things maybe you don't want to do and partial methods these are things that if you're the method if the object you're calling them on isn't in the state that you think it is when you call that these will blow up on you my team we call these Yolo methods right because that's I call ahead on a list and that list happens to be empty it's not that I get oh wait I'd you know end of the world right an exception happens and we're done so I don't particularly like those what remover can help you teach new people on your team that that's not something that you want so there's a lot of good things about word remover it's sometimes kind of painful to use the nice guys at a slam data use it extensively and have almost all of the checks turned on it's really a pan dated tool like if you're using wart remover it's probably because you're off kind of in that type level sort of mindset and you're not at all in the kind of middling it's ok to use VARs from time to time place so yeah what is this code do does it compile yes this code compiles and when I run it with something that is not an array of two elements it prints this delicious string down at the bottom because we ended up calling the two string on array which is not what you want it all does this code compile sure that compiles and we end up doing an order Big O operation and operation first to check the length of a list and then later again to reduce it we can do better than that and there's a tool over here linter that will detect these things for us so winter started with jorge ortiz when he was at Foursquare where he has moved on and done other things there's a guy named maduk maduk who's picked that up forked it maintained it added some additional things to it the documentation isn't fantastic it has lots and lots of rules that feel somewhat overwhelming it doesn't have an SBT integration so you need to understand how to pass compiler plugins through SBT to scala see to use it like I say it's a little bit overwhelming because there are just so many rules they're not terrifically well documented so a lot of times you're just reading the ED character long name and trying to infer what it does that's the downside but I found it really useful when I ran it on my projects at work it found a lot of different things like that calling to strain an array that we talked about before forgetting to close Scala sources the aforementioned unnecessary linear operation on list and a bunch of different places when i first started scala i tended to do dumb things like call to seek on a sikh where I didn't need to that kind of thing so it found a lot of good things it'll also help you stop using java converters if you're still doing that stop as soon as you can because they're going away hooray problems like I say it's got incomplete documentation it does have nice suppression via comments it's fairly slow I wish that somebody would make a list of the highest value checks that we can get for linter so that i could just turn on those things to begin with and and quickly figure out what value i could get it out of it in a new code base I wish there were SBT plugin just to make it a little bit easier to use so yeah there's my summary on winter how can we do this better yep exactly right oh I'm sorry we could call exists on sequence instead of having to do that extra is define check how could we do this better or at least differently yep we can call contains there and that'll help us find these things so there's a tool called scapegoat that will help you find those kinds of things Steven Samuel wrote that it feels pretty mature he's been working on it for a couple years hasn't done anything to it in the last six months he has a nice SPT plug-in good documentation lots of ways to suppress stuff when it comes up its rule set if I were to summarize it feels like everything I've told you about before mash into one tool which I already said that it felt like linter was a little bit overwhelming now like we're really talking an overwhelming wall of oh my gosh I didn't know my code was so bad all at once when I did run that like here are some additional things in my code that I after I'd already run those other tools on my code yeah so what I don't like about scapegoat is that it just feels so overwhelming and it has a few rules that I just disagree with like I want to be able to say my case classes final in some cases because usually it's not a great idea to inherit from a case class I got a quite a few different false positives when I tried it so consequently I say the signal-to-noise on this one is a little bit hard to deal with because it did have some more noise and yeah so scapegoats interesting it wouldn't be my first choice but I'm glad it exists what does this code do false yes thank you java thank you because equality on arrays doesn't work at all oh well so the nice folks that give us a scala test artema have a commercial tool called super safe the does static analysis it costs sixty dollars per user per year it's about two years old now they're coming up on a 1 dot 1 release the majority of things that it focuses on our problems of equality and inference it has a pretty small rule set and they have a really strong emphasis on avoiding false positives consequently they've never bothered to build suppression when I did find a false positive I sent it off to them and they fixed it right away and it's a fast tool so some of these others you don't really want in your main build this is one that you probably could have in your main build so because it's a commercial product I want to hold it to a little bit higher standard I don't somehow feel like it quite hits there I don't know at least the places I've worked it's kind of a hassle to like jump through not mentioning Oracle by name here but Oracle it's almost impossible to get you know some small vendor that you pay sixty dollars a year or two to get a tool in so it's kind of that threshold of where it just doesn't feel like it's worth it in that way maybe it's easier for you to get that to happen what is this code do yeah it gives us a lovely string when we're done of a list with three tacked on the end because anytime you do plus in this case you get string concatenation and that automatic to stringing on the left there that's not quite what I want wart remover will help you find this one so in summary I'd say go home turn on those linty switches to scala see as quick as you can I think you ought to have a coding standard in Scala style that you automatically apply my favorite ones here are work remover and linter you know keep an eye on these other things if you want to be a great member of the community go help a bite out and give it some love here is the same sort of summary but in even more colorful form I've got my slides online that you can just come grab so you know don't have to take pictures I mean but yeah I'm real sorry about that so Greenfield suggestions if you can I've got a list of Scala see switches in addition to those three I stole them from our good friend to polecat because he's amazing and I think I added a couple to them there are Scala coding standards and they'll help you you can help enforce those with Scala style and then I think it's useful to have a lint build target that you have n SBT where you use one or more of these other tools that I've talked about and there are some slow Scala squeeze switches like find unused imports that you probably don't always want to use but would be nice to be able to have you know once in a while on your release sort of thing so that might sound overwhelming I made a skeleton project that I have a PO and github that you can go grab that sets all this stuff up and and I've tried to make it easy for you to just pull those things into your project if you have an existing project you almost always are going to have to do this incrementally where you figure out a way to turn on a small number of rules maybe set some goals within a sprint you know convince your team that this has value go clean some things up and keep iterating and making progress going forward once you've done that you know you can't you can't live with these errors like the reason that sorry these warnings like the reason that we want X fatal warnings is because I've seen a lot of projects in my career where people just ignored them and you compile and you get hundreds and hundreds of warnings and some of them are genuinely bad things that you don't want and if you allow these lint tools to sort of step outside of your release pipeline they just end up being noise because if it's not on that path that gate that you have to go through to get to production people will figure out some way to go around it at least that's been my experience so don't let that happen to you if you go to the effort of bringing this in figure out a way to make it a genuine gate to the process if you get nothing else out of this talk go turn on those three switches do you have any questions for me go ahead because if you I haven't had that problem so to repeat what he said he said there's a caveat to using that set of options in the repple within within SBT that as soon as you make a variable it gets reported as a unused variable which is fatal and it won't let you do that I'd be interested to see that because I just haven't encountered that one okay so then you're saying the fix is to exclude that from the arguments that get used on starting the console okay interesting anything else thank you [Applause]