SF Scala: Max Ovsankin, Validating Changes in Typechecking on Codebases with SemanticDB
Recording: SF Scala: Max Ovsankin, Validating Changes in Typechecking on Codebases with SemanticDB
[Music] thank you very much Alexi you introduced me better than I ever could have very good marking myself so I should take some of that all right yeah so this talk is about work I did at Twitter this summer how about dominating changes and type-checking code bases with MTV yeah okay so I'll explain what all this means and I'll talk about sanic to be so let's get started so this talk is the kind of the context is automated code rewrites Twitter it's when he uses a tool called Scala fix for its code rewrites for a different purposes one of the purposes is improving code health you know just what quality of code we have migrating library usages across the codebase among other use cases so we're gonna focus on a specific rewrite of this talk which is explicit result types and this rewrite adds return types to all definitions that are visible outside the source file so as an example we have public facing number definitions it's useful for simpler static semantic analysis tools because they don't have to infer types they can just use whatever type we're giving to the definition and then this is also a cornerstone for our C which is an experimental Scala compiler focused on copulation speed that Eugene talked about let's go by the bay and he uses this explicit result types to kind of speed up a compilation however it turns out the mystery right can influence the behavior of the type checker in ways that don't show up in sources and kind of examine what that means so as an example of this rewrite we just have a simple scholar object you know it has this field bigdecimal 100 sorry and you know it's not annotated and then after the rewrite I just have like green whatever you know the rewrite adds the reaiiy doesn't really clean up whatever the type that that is that it that's just the fully qualified name but hopefully you get the idea it's basically just annotating the type so the only difference between these two files is pretty small that makes the scope of this project not to me so the problem that we're trying to solve is how we can compare potentially change semantics in two code bases beyond just syntax or sniffing right you can look at gifts regular text dibs on your source files but this doesn't tell you the difference in behavior for things that do not show up in the source files which we will talk about so you know here just a simple diagram we just have the original sources we apply this call fixer right and we have our rewritten sources so one of the parts of the solution we had is somatic TV somatic TB is a data model for semantic information such as symbols and types about programs in Scala and other languages so it's a promo debuff based format and it suppose supports an entire ecosystem of tooling so we have compiler plugins for Java and Scala that generate this they kind of you know produce a somatic DB we have tools that kind of operate on it dipping in comparison tools like the ones that I was writing and then we also have IDE like semantic analysis tools so for instance we have a code browser that you know you can see as a web application and you can click on things to go to where the defined among you know other ID features and then this also supports Scala fix for rewrites that depend on semantics as opposed to just syntax so maybe the rewrite needs you know type information or something similar so as part of our tool chain we use semantic TB Scala C which is the compiler plugin for Scala that generates somatic to be as mentioned earlier so here's just an example of what somatic to be looks like this is a printout from one of the semantic devtools which is I think it's been a P that prints you know just shows what semantics to be looks like ok so here I just correlated what we see in somatic DB with the actual source file so we have the same source file as before we have the % object which semantics to be sees as you know an object that extends any ref we have the field that it is declared we have kind of different sections of the somatic to be file right we have like a symbol section it describes whatever symbols we have it has occurrences which have a line number they have a line number and then whatever kind of that piece of the file is referring to like the actual symbol we have a convention for how to uniquely identify symbols that can be seen outside the source file and then finally we have a synthetic field which is going to be those semantics that I talked about that you know don't actually show up but they do affect the semantics of the program so in this case we have big decimal you know when you write parenthesis like that it's not just big decimal as a function it's actually an object with an apply method so that apply gets added by the compiler so for the purposes of this talk Scala synthetics it's an overloaded term but here just means fragments of expressions but the compiler inserts that the user didn't write and this is actually you know somatic behavior that we cannot catch with the source disk right so this includes expansions infer types macro expansions and for loop disagrees so as an example you know we're calling the important part here is we're calling last on an array and my default areas are not actually collections so scholar wraps it in this red area ops thing that turns it into it wraps like a wrapper around it that supports the collection operations and then that this is basically what we see in this meant to be so semantics to be has a parallel model for Scala synthetics and we added basically a tree protolith message that describes these synthetics so what a synthetic is in a somatic DB it has the source position and then it also has the tree that describes what fragment we have so like for the previous example we have the source position from 69 for 69 to 34 and then we have the original text and then also you can see ref Area ops of string around the start which means we're wrapping the original text in this mythical and you know just to summarize it's basically a standard typed abstract syntax tree so to call this Rothery ops we do an apply on a type of play on pre-doctoral cherry ops and we apply it to the original tree so now with the tools we talked about we run semantics DB or we run the Scala compiler with the semantics to be plug-in on both code bases and this produce two sets of semantics DB that are parallel to the sources that they were running and it includes synthetic information right which is the information that doesn't appear in the sources but is the information that we actually care about in terms of semantics to think so the next step is we just want ad if this right so the next thing we did was implement a command-line tool called Benedict and it works very similar to just regular text command line diff so it doesn't show any output if there's no diff otherwise it shows text output the describes what are the semantics that have changed based on the somatic DB so for the context of meta diff what it does is it looks at synthetic trees at different positions and if at a certain position the synthetic trees are the same then for our purposes this means that the somatic behavior must be the same for them so this is how we're able to basically catch if there's any changes in somatic behavior so the logical step would just be to run meta diff on these two sets of somatic to be but this actually doesn't work because you know if you consider code that is written before and after the explicit result type sorry right for instance the positions are all wrong right because we added snippets of code like all the positions that somatic to be uses to describe the semantic information I just are off so if we just compare the original files the original somatic DD files it doesn't even make sense it's telling us the wrong differences so what we actually need on top of this is a tool that finds correspondences between the positions which we call a reconciler and our implementation just shifts back the positions in the new somatic DB so that the maybe there are differences in semantics but at least the positions are the same they correspond to each other right and we can do this because the rewrite is relatively small it's relatively local so that there there can be other semantic implications but for most pieces of code this works fine so finally here's our entire tool chain we have you know we have the original sources we compile somatic to be plug-in and we have the new parallels Magda bees we run the reconciler which takes all the previous information we have and then it produces written new source somatic DB that is more similar to the original source semantically being in that it only describes changes that we think are semantic changes and then we run met ative on that and that gives us our semantic differences so finally we've been able to capture you know any differences that haven't showed up in the code between the two source trees with this entire tool chain so we are open source the two repositories of note are Scala meta and RSC Scala meta is kind of the project that semantic DB is a part of that you know there's a suite of meta programming tools for Scala and RSC is you know the compiler I mentioned earlier the Twitter's developing yeah so thank you very much ok so the question was where are some examples of rewrites aside from adding exclusive type annotations I guess for the scope of this project mainly because the reconciler is very dependent on the behavior 30 right we focused on this specifically right as for other rewrites I guess that question applies to Twitter more generally I'm actually I don't think I can recall any other rewrites that we were using like there they there are applications but I'm not quite sure maybe Eugene can help answer this question I would ask the question so we're all you know going to see the finds RIT and Stella community about transition to dotty Scala 3 right so people wonder how it will affect call basis you know Verizon has two code base and companies like this so we all need to lean so how can this help us in transition - yeah so the the tooling around semantics DB so the progress as far as I know is that it works for Scala 2.11 2.12 the nice thing about semantics DB is that the format is general enough so that it can express you know any version of Scala that we give it in the future I don't think that there we have implementations that plug into dot e that actually spit the semantic information right so once we do have that semantic information then we can use tools like this and this will tell us basically based on our model of what we think semantically be like what what the semantics that semantics to be expressed is that model will be able to tell us you know if there's actually any differences in your code after you've migrated to dotty so you could definitely apply it or something like that [Applause] you [Music]