SF Scala: Seth Tisue, Tools for Taming Tangled Code!
Recording: SF Scala: Seth Tisue, Tools for Taming Tangled Code!
[Music] hi so this is a new talk this is the the first time in that I've done this so please don't hesitate to ask questions make suggestions you know either during the talk or or afterwards I've been doing Scala 10 years now I've been on the Scala team at light bends since 2015 and I've been living here in San Francisco for for about a year and a half my qualifications for for talking on this on this topic it's that this is gonna be based on two experiences that I've had in my career one is that for for many years for for more than a decade I was the lead developer on a large initially all Java later mixed Java and Scala code base so this was already a legacy code base that I inherited even when I started and and then I had to you know keep it alive over a course of many years so I died you know hope that I learned something about how to improve the health of a codebase over time and the other experience that this talk is based on is that in 2016 I built with the help from some of my teammates that like bend a tool called sculpt and I'll explain a little little more later what that is there's two possible ways or two possible reasons this this talk might interest you one is that you know any any solid programmer is using tooling this is this is a talk about tooling I'm I'm presenting will see as this develops that I'm presenting an area where the tooling that exists for Scala is is not that strong there is there's some but it's not like super well-developed super solid so I also hope to inspire people who are building tooling either for the Style community or just for use on your project or within your company to provide some some inspiration on that so taming tangled code we all like clean code we'd like to achieve clean code clean code of course looks nice it's well formatted but it should also be well structured at a deeper level not just at the level of does it read well line by line and and method by method there are different things we might mean by by well-structured code I'm gonna focus on a particular meaning of that which is the internal dependency structure of code this is so when I say dependency I'm not talking about external dependencies I'm not talking about libraries you're pulling down and throwing on your class path this is about your code and so when I say dependency I'm talking about one place in your code that depends on that refers to another and and that does it at compile time so this is all about compile time tooling our goals in in wanting to improve the the dependency structure of our code our modularity which you know I hope III won't unpack too much why that's good but it has like all sorts of benefits this is you know so many modular code is more readable it's more comprehensible it's it's more it's more maintainable and it's also it can also serve different purposes so often the the motivation for improving the modularity might be of your code might be to split out some functionality so you can use it you can use it elsewhere even if you aren't doing that you might like to improve the dependency structure of your code just because you'd like it to compile faster and that was actually the I think the the major motivation for the customer who funded this work at light Bend incremental compilation in stralla is driven by the dependency structure of your code what what if I change a piece of code the stuff that has to be recompiled is everything that that depends on the piece of code I just changed recursively so a simple picture of what's what's good and bad and in dependencies the the best situation is that you have two pieces of code that are entirely decoupled and can be compiled and developed completely separately next-best is a one way dependency and the thing that we really want to that we really want to avoid is circular dependencies and circular dependencies are a bigger problem the the larger the grain size of the code that we're talking about is so two functions that mutually dependent that's probably fine to files to source files that have that have circular dependencies between each other getting more questionable on any project I've ever worked on worked on I've never allowed circular dependencies between packages at all I think that's a like a really strong smell and and we know that at the binary dependency level that circular dependencies are are just impossible I mentioned that I worked on a large tangled codebase for many years I was unable to dig deep enough to get like a diagram of the dependency structure that I started with but suffice it to say that basically everything depended on everything else you know directly or indirectly it's it's remarkable how how few it's remarkable how few bad dependencies can create a situation like that you know you can have code that's kind of mostly modular but then there's that one place where this class just reaches into this other class and you know there's some math and in in graph theory though that will tell us that you don't need to do that very often before before you you literally have the situation where everything is is mutually entangled so this this diagram on the right associated 2013 I always show people this diagram not knowing if they're going to be impressed or horrified like oh that looks nicer oh my god that looks you know what wow what a mess I'm encourage you to see this diagram on the right as good compared to compared to where we started so each of the each of the nodes in this diagram is a package that there are no cycles between packages although some of the details in this diagram can get messy there's a clear overall macro structure this is layered layered architectures separated by the dotted lines where each layer has direct dependencies only on the on the level below if you look even more closely at this you'll see that there are there are clusters of nodes that have no incoming no incoming arrows so that's great the there's whole modules and collections of modules here that no other code depends on at compile time you know all this code is working together at runtime but I can it you know anytime something has no incoming arrows I can recompiling it has no I don't need to to recompile anything else and the this structure was actually enforced by our but by our CI so you you could not you know check in something that violated the structure without without a some automation complaining at you there's a danger when you're doing this kind of work that you you know lose track of that it becomes all about boxes and arrows on paper and you're just like architecting and and and and forget what it's about but the the modulation that that we did on this code base was what was actually about about business goals and about features so that there were there were features that we wanted to do that required the modules to be separate we had an authoring environment that needed to be separated from a runtime engine so the runtime engine should be should be delivered could be lightweight and embeddable we had this began as a GUI application we wanted to support command line operations so all of that code had to be had to be disentangled and then we wanted to hit to make modules so we could open-source them although actually the whole code base is open-source now and and we wanted to run some some of the code in the browser on Scala Jas so that that required making that code completely independent and removing all of its getting all of the Java code since you know you can't just throw Java code into a scholar application so we had to make it pure Scala so that these are the this was really this kind of work can actually be kind of life or death for for for a code base like it can it evolved to to suit new needs you can do this kind of work of disentangling code by hand certainly but it's really a lot easier if you have some tooling to help you and there's two possible approaches that tooling for dependency analysis can take you can have a tool that looks at the bytecode that comes out of the compiler and looks at the dependency structure of the bytecode or you can analyze the source code and and try and pull the dependencies out of that source code analysis is I think definitely preferable there are particularly if you're interested in reducing compile times so there are you can have dependencies in your code that the compiler sees that don't actually leave a residue or in effect in in the bytecode so you could fix up the byte code without fit without actually fixing all of the all the problems that are that are having tangles in your code so I think ideally we would prefer source code analysis but byte code analysis is pretty good and it has the great it has the great advantage that it's language independent so a you know tools that work on Java bytecode but most of them work you know as expected on the bytecode that comes out of the scout the Scala compiler even if the authors of that tool weren't weren't anticipating that so I'll show the the bytecode tools that I'll show you know come from the Java world but just happened to work in Scala so these these two tools for for bytecode analysis one of them I think almost everyone who does stuff on the JVM has heard of and that's ProGuard the the other seems to me to be quite obscure so I don't think I've ever run into anyone who knew what it was so part of my goal with this talk actually is to raise awareness of this tool class cycle which I found incredibly helpful in my own work on my own code but but people don't seem to don't seem to know about it both of these are very very mature stable tools that they haven't changed much too in recent years but both of them were updated to add support for Java 8 and for lambdas so you know there's there's somebody there keeping the lights on and and and making sure that these things keep working even though they even though they aren't changing much anymore and then we'll come back to the for the the tools the first source code analysis I'll show a tool by Holly Lee called a cyclic and then sculpt which is the the thing we built at-at light Bend so by doing source code analysis I think we can expect higher fidelity but the the tooling is less less mature less less finished okay so so class cycle this is an open source tool developed for Java I I never never had a problem using it on Scala bytecode it always works fine and what it does is it's just a just a jar that you point at the the directory or jar that has your class files and and it analyzes everything anywhere in the class file that that refers to that refers to anything else and then collects that raw data but actually sure if it makes the raw data available what I always used was was a report that it can that it can generate and this I put a screenshot on here in case I wasn't able to show the user interface but it's better if I can actually show the UI live so what you'd get out after you after you run this on your code is there's a few sections here there's some summary statistics at the top there's a list of cycles so these are I said I said in my code that that package cycles are forbidden but but class cycles within the the same package I think are usually okay so we've got 54 class cycles and it will show you you can see what the classes are in it and and get some other statistics and and and zoom in layers okay so this is helping you understand that the structure of your code by but by grouping in the layers layer 0 those are the classes that that that can be compiled purely by themselves without reference to to any other code level wants so then once you've compiled level level level zero what else can you now compile that's level one everything in level one refers only to to level zero classes or let's see this layer one layer one are you allowed to refer to other layer one classes I think you are and so forth so and again this is this is something you can zoom into you can see what all the what what all the classes are in a particularly ER so at the bottom we see lots of these are usually interfaces and little utility classes and then when we get all the way up at the top at the top at the top level it's like the the big you know master class that glues your whole your whole application together and then so that yes sure so the the question is about the is about granularity if if this is telling me that you know class a depends on Class B can I get in any more detailed information on on where exactly that that dependency is coming from like is it is it in this method so let's let's use this last section and and this is the the big section that goes all the way to the end of the page that just just lists all of your packages and all of your classes so we can look at the agent class uses internal and it looks like this doesn't give you method level granularity I'm a little hesitant to say for sure that it that it doesn't but I don't I don't remember using it there was yeah I did - to make sure I'm answering that question correctly I'd have to I have to look it up afterwards it's I don't think it's in this UI but I think that command-line tool might actually do it and I'll show the command-line tool also okay so that is class cycle and in the interest of time I should move on and just show you one more thing about class cycle which is that it's also exists as a as a command-line tool so this is something that you can actually make part of your build it if you want and this this is how we actually enforce this structure in the CI on this project so to use the the command-line tool you you write a config file that has a bunch of rules in it and these are a few sample rules the the key words here are our independent of and directly independent of so you can specify that you know the this pet this package is it can depend indirectly on on something else but it's not allowed to talk about it directly right so that that would be directly independent of and then independent of is the is the the stronger predicate which is just like no you can't have you can't have this stuff anywhere and your in your in your recursive tree of stuff that you depend on I'll just I'm including ProGuard just briefly it might be so you know pro-gard we mainly think of it as a jar shrinker and optimizer but in only in order for it to do that work of shrinking your jar that this analysis of dependency structure bytecode it's exactly what it's doing and it's really good at it you know program just this thing that everybody uses it's been around for years it's like it's it's it's bulletproof so what weather ProGuard is a you know possible basis for for more sophisticated tooling in this area I don't know but III have have definitely used it as a backstop like you know okay I think I've got things modularized and now I'm gonna tell ProGuard to make a jar that just isolates that slice that isolates something and if there aren't any dangling references in that jar ProGuard will will find them and complain it's really good okay any questions on the the bytecode before I changed gears and start talking about source analysis okay so the the source analysis tools that I'll talk about our Scala specific the first one is in this space was was built by Howie Lee who I think people have been at SF scholar for a few years anyway know well and that everyone involved with open source college knows well so the this is something that he built for himself and and uses in his own projects so it's it's a it's based on similar technology as sculpt that we're going to see next but it has a much more it has a much more focused goal which is he specifically zeroed in on preventing cycles in your dependencies it's a compiler plugin it operates by putting this this special magic import in the source files that you want processed or you can also just just say process all the source files and you specify whether you are happy with the loser rule where where cycles are only forbidden at the package level or whether you want to enforce something much stricter and and disallow cycles even at the even at the level of source files I don't think it does class level and I think that's that's probably sensible like I'm not sure you would ever it seems perfectly normal to me to have a cycle of classes in a single file and if your code when the plug-in runs if it does find cycles then you get a nice a nice detailed report right down to the level of the particular line of code that each line of code that's actually participating in the cycle that you find so that this is this is a terrific tool but it is somewhat special purpose okay and now I'll talk about this this thing sculpt that we built at light Bend so that this was this was developed for a customer who wanted it but also I think we always assumed and expected that we would make it available as open source as well so I did most of the work on this but a Adrian Moore's you know did the the most difficult of the compiler wizardry the that's in here and then Stephan Sagar also on the scala team did some of the coding as well so we just open sourced this last month and you know i i'd put the i'd put like a maintenance status slide last but i think i'm gonna cover this now actually just so it's just it's you know so you you know what you what you're getting when i start describing the technical details so that this is not actively supported and maintained but nor is it is it completely abandoned where it's somewhere in the middle so there's a there's a key component here that i'll talk about a little more detail called extract dependencies that actually generates the the raw dependency data from the compiler that piece is very much not a abandonware so that that's a piece of code that was you know was hard to write it what would have been difficult for someone outside our team to to write it I think without a lot of study and it's a it's like a possible nucleus or kernel for all sorts of possible tooling that you could build so that that piece of it if anyone wants it you know it's like if we're not going to be hacking on it unless people step forward and say you know I want this and I'm willing to help hack on it and test it but if people like that do appear then what will work on it with you the rest of sculpt is more like here it is m'm sends a pull request with a modest fictional improvement I'm happy to review it but but you know we're not going to be doing any any further major development on it so that's that's the that's the status yes we'll know so in order to build it you have to you have to build it from source yourself I I don't think that's a big hurdle it's just a standard SBT thing it you you know clone the repo type SBT published local and and you're good if there's if there's demand to add publishing I you know I I would merge a pull request that added that that that added publishing Oh III didn't repeat the question so the question was whether the whether binary artifacts have been published okay so this is a compiler plug-in like a cyclic it's a it's informational only so that there's no attempt to enforce any particular policy we're just producing data that you're that you can do what you like with the this kernel that I mentioned the the light the piece of code that that's the the key to the whole thing is actually code that that we borrowed and adapted from zinc if you don't know what zinc is it's the it's the incremental compiler component of SBT so it started life as within the SBT codebase and then it later became available and published it as an independent artifact so that maintainer zuv other build tools can add incremental compilation to their tools as well so you know the maven plugin for Scala uses it I think pants and Basel both use it it's it's it's the incremental compiler so the fact that that we're using the same code both in zinc and and sculpt means that you you you can trust that the exact same information that you'd bet you get from that that you get from sculpt is is the same information that that guides incremental compilation and so that you know that's should be especially appealing to you if one of your main motives for this is wanting to reduce incremental compile times the and I've put the source file I put the URL of this this kernel source file it's some some pretty hairy compiler code I I say that to tout its value but I also don't want to scare I don't want to scare off anyone who might be interested in trying to tinker with it if it needs fixes you know it's it's it's doable the there's no most of the structures there and it from this point forward it should only need the most it should need is tweaks around the edges so what does sculpt add beyond just this kernel so that this extract dependencies code listens to the compiler as it's working and it the raw dependency information that's coming out of the compiler is its first stored is in memory as instances of case classes and these are these are very simple case classes they're just data they're not they have no references to internal compiler data structures they're not in the compiler cake it's just data and if you if you'd rather have it on disk in order to deal with it there's also code to to output that information as as Jason and what we'll see in a second what that what that Jason looks like so that one possible use mode for this is is just the the you run it in the mode where you request Jason output and then you do your own analysis of the of the data that comes out we do have some fairly simple analysis that that the tool can can do itself oh I should have said you can request for the you can get the raw data at the method level or you can you can ask for it to be aggregated at the at the class level those both of those options are supported I showed you the in class cycle the layer report and the cycle report sculpting can do that as well you know we copied that it doesn't come out as a nice HTML page it just comes out as as text but the the logic for for computing that is there Aleksey how am i doing on time okay perfect excellent and okay so let's look at some sample output and I'll switch over to the browser to show that so the examples this is the repo on github Scala sculpt and I think well if I do say so myself that I think the documentation is pretty good like to the the code that that exists and the capabilities that exist are fully described in the in the readme I won't even have time to show you everything but I'll pick a few samples here so that this is an example make the type a little bigger yeah make the window bigger too okay so this is what the raw dependency data looks like in Jason form and this is let's see this is at the method level right so we're seeing entries that have things like definite right so this is the this is the constructor the there's this represents the dependency graph there's there's two types of edges there's extends edges and and uses edges uses covers all other kinds of references that you can have and we see that this includes dependencies on on external code as well so it'll it'll show you what you're using from your whatever's on your class path including the the Java and Scala standard libraries the method level data is voluminous and you know hard to deal with without tooling which you know it doesn't currently exist so I think it that you will most often want to use the option that that aggregates the the the data at the class level and then maybe if you became interested in a particular edge then you might flip over to the to the method level data to see you know where that class level edge is is coming from okay so I don't think I'll skip over this the stuff about the case classes and memory yeah so here's what the here's what the layers and cycles reports looks like so this is this is the layers report for a really simple piece of code that just just where's the code for this this is this is just to two object definitions that you know that refer to standard library stuff so all the standard library stuff ends up in layer 0 and then your user code is in layer 1 this is an excerpt from the layer report that I ran on on a code base of my own and oh let's see so the one on the bottom here this is the layers and this is this is what a cycle looks like so this is this is saying I have these eight classes that that are all mutually entangled and here's the here's the name of the names of those classes there's there's some stuff where you can modify the you can actually like delete edges in memory and then see how the report changes I'm not sure if that's if that's used for not question from well the the question is does this pick up on things like like F bound polymorphism so you know because this is source code analysis and not bytecode analysis it will it will pick up stuff that's only in types is there something special about F bound polymorphism in particularly curious about or is it just it's like stuff that that's old that only exists in the types like Oh does it does it classify the relationships no yeah okay so I'll flip back to the slides now of course there's all sorts of directions that a tool like this could go and all sort of all sorts of features that it could have or should have I'll just mention that the most obvious missing features in my opinion at least is that you I think you you would really want something that can deal not that can aggregate so method levels the raw level we can a grated class level we don't have code that supports package level or source file level aggregation I think if someone wanted to add to add that you know it really wouldn't be wouldn't be terribly hard and I said this already but I'll I'll I'll say it again because I think it's important you know the anything you want to do with this data you you're not up in compiler guts it's it's just it's just simple data okay so we covered the maintenance status of sculpt are there any I should pause here there any questions about about sculpt in particular yes okay so that the question is does that does having list your build is this gonna slow down your build if you enable it so I have to admit that I haven't measured that I would expect that any slowdown would be negligible so or if not totally negligible III I can't imagine how it could possibly cost you more than like 1% or 2% or something like that I would expect it wouldn't even be that that high it really the extract dependencies code is really not doing can't say none it does very little additional traversal of your code it's mainly just recording information that already exists within the compiler it doesn't have to do a lot of work to get this any other questions on sculpt yes will okay yes great great question so the the the the question is if I what happens if the if this plug-in is enabled when I do an incremental compile you know is it like what what what happens so the the answer to that is you know don't do that that that's not the expected use mode the the expected use mode is that you would generate this data once from a clean compiled I I mean it's it's a question I hadn't considered before and I think it's I mean it's entirely Paul I would expect that you would get accurate information out of out of the follow-up incremental compiles that you would just then be faced with the problem of how to combine the data that you got from the full compilation run with the new data that you got from an incremental compile you know the the logic for combining those doesn't exist but I don't in practice I don't I I think this is the kind of thing where it's kind of like development is paused you do a full compile you get this information out and you go and you'd often analyze it I don't envision this being used in in in tight incremental coding cycles yes well I see you know that that's I'm really glad that you said that because that's okay yeah so I get it now and actually yeah that's a great use case for this why didn't I think of that so let me let me repeat what we'll said so you know in the course of this talk I've been in envisioning this as a tool where you're looking at your code base as a whole and trying to you know make improvements to the dependency structure but Will's question is suppose I I just it's like I changed stumps I changed something in my code for you know in a play routing file for example and this really long incremental compile followed and that was unexpected and I'm unhappy about it and can this tool help me analyze what happened I I would think that I would think that the report that you would get out of the out of the plugin on that incremental compile would like yeah it would have like only the information that you wanted great idea I'll have to have to add it would have to have to actually be tried and in practice but I think that would work in that might save you quite some effort sort of disentangling of the part that you care about from from the big picture anything else on sculpt okay so we're almost done I would just want to mention a few more tools that exist for well if well okay so to wrap up I'm still tie I guess oh I'm not there we go okay so I've present I've presented this thing that's that's incomplete I think it's possible that that some projects might actually find it usable and useful as it is I'm not sure we just open-source this last month so we don't have you know information on that you know there's been interest but I don't know if anyone is actually out there trying it but even if even if you you know this is like too primitive for what you want to do I'm you know also putting this out there in the hopes that it could be a basis for either building your own tooling or you know either for your own use or for use in the community okay so a couple more tools to mention and then I'll be done these are tools that I actually don't know very much about so IntelliJ includes this thing called the dependency structure matrix that has anyone used this can you raise a hand if you've used this okay it's only it's so this is a commercial feature and it's only supported for Java which is why I won't spend too much time on it it looks pretty amazing or pretty powerful and also potentially incredibly confusing right that there's a whole manual section in IntelliJ that explains what all this stuff is it's like the Rose or this at Tom's or this you can expand stuff it looks like a great tool for really digging into digging into the structure of your java code so and I'm including it you know on the off chance that you're maintaining a Java code base that you want this kind of thing for I also mentioned it just shows you know if you're building this kind of tooling it looks to me like you could look at this and get a lot of ideas for for data that a like sculpt could be could be you know analyses that it that it could be doing I'm not totally surprised that no one's used it though okay and there's a we found out after we build sculpt that there had been an earlier project called extractor that looks very similar I haven't had time to to look into it it hasn't been touched in the last two years so this is probably only of interest if you're if you're interested in building this this kind of thing yourself okay some some closing thoughts on how to keep your your code clean once what once you've cleaned it up with these with and made it more modular with these techniques the the biggest hammer that you can use is to move code into separate repos you know at that point reintroducing these compile-time dependencies becomes impossible the slightly smaller version of that hammer is making sub projects in your build tool and and that that also makes unwanted dependencies impossible so you know if I tell it if I put stuff in separate source directories and tell SBT you know which sub projects I expect or not just expect but allow to depend on each other then code that that goes outside that structure just simply won't compile so it you know if you can do that then then you're good from from that point forward if you do make a tool like last cycle or sculpt part of your your build on ongoing I'd strongly recommend doing that as part of CI I had a great experience with that and an entire cycle and what one closing thought if the if the the tooling that you want doesn't exist consider building it so the you know I said I had this great experience with with cycle but actually it really didn't didn't do exactly what I wanted and I could easily have given up at that point but I did some custom work on actually doing doing cogeneration on a class cycle config file so the like that the config file that I wanted involving recursive dependencies between packages like writing it by hand would have been essentially impossible but it was it was it was not that hard to write a little code generator that should expand the rules I wanted to write into the the rules that that the the class cycle can write and even though even though the code that I built for that you know was only ever used on this one project it was was was totally worth it okay so I have a couple questions for you in the audience are there you know when when we were doing the the preliminary research on sculpt we tried and tried and tried to find out about everything that's out there that's kind of in this space did we miss anything that does anyone here know of something that like what a lot but logically had a place in this talk yes well J Depp's I've never heard of J Depp's what can you tell us about it okay so this is this is the the former project jigsaw are we talking about yes that's totally relevant and the reason I didn't include it is that I know almost nothing about it so but that yeah that that's a great point like there should be a slide on this like you know jigsaw has has implications for this I mean so right so you know I do know that that jigsaw actually involves specifying metadata for your modules that that like specifies what's exported and what and what's not so it's like more levels of access control beyond just whether something is public or not and and there's enforcement like you can if you enable the enforcement then you know code will will refuse to compile or refuse to use refuse to run if it if it violates these module boundaries that you set up so so yeah thanks for bringing that up that's totally relevant class cycle yeah it's um it's a little hard to spell it's like how many where the esses and C's and in that it's here so that's that's the that's the URL it's on SourceForge which tells you how old it is but it was updated in 2014 for Java 8 and yeah I really recommend it okay that's all thank you [Applause] you