Scale By The Bay 2021 : Anatolii Kmetiuk, Troubleshooting Techniques for Complex Codebases
[Music] great thank you eric um thank you for thank you everyone for coming to my talk hello everyone so the talk is about debugging the complicated code basis on the example of said dottie codebase which is scala cere compiler and before that i wanted to briefly say a few words about myself but eric has already introduced me very well so i don't think there is much need in that my name is anatoly and i i have been working for the past two and a half years as a compiler engineer at slam ptfl so i have been helping out with with the scholar 3 effort and one of the challenges that i am trying to solve at work is a problem of how to effectively tackle issues so as in any big project as in any big endeavor there are bugs and there are um issues and especially when we launch as a compiler into the into the public like release a big major release there is a lot of issues because everyone starts to use it so the question is what to do with that and how to effectively tackle that and basically this is what my talk is all about it is about my findings of how to make your workflow of fixing issues more efficient what i found when i was trying myself to see how to more efficiently fix issues in the compiler so this stock is focusing um specifically on the dotted compiler but the techniques techniques for debugging i believe can be very applicable to the general case of complicated and complex code bases and the mindset with which you approach this issues is also generalizable to big code bases so what the stock is all about so in this talk i would like to highlight and introduce the repeated patterns that i found in the issue tracing workflow and also i would like to to show how you can reduce the cost of experiment so what is the cost of experiment well when dealing with issues you need to experiment a lot an experiment is for example when you produce an issue on your machine or when you make a hypothesis of what is causing the issue and change something in the compiler in the programs that is causing the issue and reruns a program to see if the issue still manifests itself an experiment is also when you runs a compiler with some debug options and so on and so forth so why is it important to reduce the cost of experiment well the cost i define cost as in time cost and effort cost well if the time cost of an experiment goes down you obviously solve issues faster but also the less obvious thing is that if the time and effort of your experiment is low it means that you are able to do more experience experiments because they're cheap what i mean by that is that if a cost of experiment is high like for example it takes a lot of time to rerun the compiler you're going to sync twice before actually doing that because it takes so much cost and this means that this means that you're going to do less of those experiments and gain less of the insight and this is why it is important to reduce those costs not only to gain time but also to make sure you experiment enough and collect enough insights and nothing is blocking you so here is a plan for this talk we are going to cover those concepts first i'm going to introduce the essential parts that goes into the process of debuggings compiler then i'm going to introduce the techniques for addressing those each each part and finally i'm going to show you a little five minutes demo of how my setup looks looks like in practice when dealing with actual issue alright so parts of troubleshooting phases of troubleshooting what are same i distinguish three phases of troubleshooting and the first one is issuer production to effectively be able to solve an issue the first step is that you need to run it in your local machine because that way you're able to play with it once you have reproduced the issue once you have confirmed that it is happening on the latest version of the compiler on your local machine next step is to figure out where exactly in the code base it takes place and that calls this step navigation because you need to figure out what exactly the compiler is triggering the issue well in the process of navigation well usually the way it looks like is that you go to the immediate manifestation of the issue in the compiler you see what causes this manifestation you you go to the site of the cause of manifestation and you figure out what is causing that and so on and so forth so the trick is to follow the causal chain of um of causes that contribute to the issue to figure out the root cause and the causal chain can be pretty long it may may involve like 10 links dozens of links so navigation navigating those links is important but once you are able to do that the third step of troubleshooting is what i call sense making well you need to make sense of what you see because you're going to like while navigating through the causal chain of an issue you're going to see a lot of code and you're going to need to make decisions decisions on whether a particular piece of code is important or not on whether the compiler behaves right or wrong at a particular point on what might contribute to this a particular point going wrong and so on and so forth and to make those decisions adequately you need to make sense of what you see you need to understand what you are seeing all right so these are the three parts that go into troubleshooting now let us talk about how to like what what you can do to reduce the cost of going through every part on an example of my setup so i'm going to show you how i set up my environment based on sublime text but i would like to you to focus not on the particular editor but on the concepts on the capabilities that the editor provides me to work with the issues because the truth is that you can write those capabilities yourself if you know how to script your text editor which is basically what i did for sublime i scripted sublime editor so if you understand the concepts that the thing is that you can easily script your own editor to support the same scene for your particular case all right so the first step is issue reproduction the first phase is to make the issue run on your machine so if your production is what you need to do for every issues that you encounter this means that issuer production is something that repeats over and over again through your workflow through the time that you are working on the project and one thing that i have discovered when working on complicated and complex code bases is that if something repeats then you will benefit a lot from standardizing it standardizing it means that you impose a rigid structure on how you organize information and rigid procedures that you follow to well make sense happen related to your workflow if you do that if you really standardize your workflow that repeats from time to time what you gain is less room for error because if you do things the same way over and over again and the way is well defined there is less chance that this way will go wrong why is that important well because when fixing issues in complex code bases there is plenty of stuff to go wrong and if you can remove some of the stuff going wrong you will already gain a lot of time and you will already gain a lit a little bit of recent space for you you'll be able to spend time not debugging your setup but debugging the issue in question so what is my standard procedure for organizing my issue fix my issuer production flow well you see it on this screenshot so basically it's not very complicated it's all about having every issue in its separate folder and having all the issue folders in one playground folder so called issue workspace folder every issue folder obviously contains the scala file with the issue itself that i'm working on but also one thing that is specific to my setup is that it contains also say issuer production script and the issuer production script will cover in the next slide but before that i want to mention that the issue in question is a compiler related issue which means that success color filing questions that reproduces the issue the idea is that you're going to compile it with the compiler and this compiler is going to do something wrong and your job is to figure out what it did run and why issue a production script what is it and why do we need it in the process of fixing issues well basically an issuer production script it has an extension iss which doesn't stand for international space station it states it stands for issue and basically what it is is a collection of sbt commands so think of it as so uh initial production script is the same simple as pt as a bash script to bash which means that it is a list of sbt commands that spt is going to execute sequentially once you run it from the dirty project so all of those commands that you list in the issue reproduction file will be run from the doty compiler project and what this particular command says on the screen is basically it is a compilation command if you run it from it's a doty project from the compiler svd project what will happen is that it will try to compile the bad dot scala file the only thing that is not svt specific in this command is a dollar here variable so dollar here is specific to my setup and it points to the directory in which some issuer production script lives and there is a plugin there is an sbt plugin that will know how to execute the script step by step uh to the right at points 2 and 0.3 you see how this um how this is your production script is launched from sbt so the sbt plugin adds a new command to spt which is issue command so you write issue then you write the name of the folder that with the issues that you're trying to execute in this case it is demo so you write issue demo and then it copy paste essentially all the sbt commands from the issuer production file too to usb console so the question is why why do we need this issuer production script at all well my motivation for doing that was twofold so first thing is that sometimes sometimes you're producing an issue involves several commands for example if it is issues that happens at runtime there are two commands one to compile an issue and another to run what you compiled and doing that type in those commands from sbt every time you want to reproduce an issue is tedious the second motivation for me to create this plugin was that the way you work with dot even debugging it is you you utilize the debug flex very extensively so doti scala 3 compiler has a lot of flags that can be used to provide you a certain debug output that tells you what the compiler does so stuff like stuff like the output offset trees that the compiler works with during the various steps of the compilation stuff like some advanced information about those trees and stuff like stack traces on errors which point you to the place in the compiler where error happens and so on and so forth so this is a lot of flags and it's not very convenient to work with those flags from the sbt console because it's it's tedious to add flags it's tedious to remove flags those flags are in one big line it's tedious to jump to various flags in the console and so on and that's why it's more convenient to write these big compilation commands into a separate separate file and send point sbt to do to execute that file well with that motivation in mind we can look a little bit in details on the current slides that i'm showing to see all the features that the issuer production script syntax provides so first first point is it provides bash commands capability so not only you're able to write sbd commands you're also able to write bash commands this is useful for example if you want to clean up the output directory as is shown in this slide this documents clean up the output directory the second point on this slide is the commands for obvious reasons and the comments are particularly useful when enabling and disabling various flags so that you don't need to erase a flag you can just comment a flag out which brings us to the third point well if you want to comment out the flag you better have it on the separate line right because it's much more convenient to comment out an entire line than comment out a chunk of a and for this reason the spd commands in this plugin are can be multi-line so if if you so you can continue writing an sbt command on another line provided that you indent it if if a line is indented it's treated as a part of the spd command on the previous line which is which enables you to for example list every flag that you need for debugging on a separate line and then turn it on and off selectively via commenting finally as uh first option is the first the first point on this slide is a showcase of uh spt commands per file which will be executed sequentially which is nice to have when you have several commands to reproduce an issue and well finally the fifth point shows showcases you what happens when you execute this command so output the entire output you see to the right is what is the result of of execution of this entire script great so we have covered the question on executing on executing on reproducing some issue the issues with what they used to reproduce the issues and at this point maybe it is a good idea to break and ask for questions are there any questions eric um no we don't have any questions at the moment okay great is that like 10 minutes left or 11 minutes okay 10 minutes okay so um then i'll quickly showcase the two other steps beside reproductions that we have so first step so the second step after you have managed to reproduce an issue is navigation where exactly it happens in the code base here is an example of something of an error of a compile time error we are trying to assign a string to an integer obviously it's an error the question is where exactly it happens in the compiler what triggers it well to figure that out a good idea would be to enable the trace error side flag which is by the luck error flag what it what it does is um whenever you compile something and it produces an error there will be a stack trace that points where this error happens in the compiler now we encounter a problem because we want to conveniently jump from stack frame to stack frame and this is problematic because my editor doesn't know how to do that but since there are dozens of frames to jump through it's tedious to do that by hand fortunately my editor is scriptable so it's not hard to teach it how to do that so that's that's just what i did i wrote a little plugin for sublime that allows me to jump to a stack frame on alt click which saves a lot of time when you want to explore dozens of frames and that was all i needed for navigation part as a final part that goes into issue tracing issue debugging is sense making all right you have those dozens of frames you're able to navigate through them easily now you need to understand what's actually happening there to be able to see what's going on wrong and to do that the primary tool in your disposal is examining variables normally it is done with a debugger but that it doesn't have a good support for debuggers you can make it work if you really want but it will not be a very good experience so i'm not using that in my workflow what i'm using is print print statements but it's tedious to print variables every time by hand and so and so what i did is i instrumented the sublime editor excuse me can you hear me i think something happened to my mic uh yeah we're able to hear you okay good so um yeah and for this reason i instrumented my text editor so that i can select a variable highlight it then i can hit a hotkey and it is going to turn that variable into a block and the semantics of the block is to evaluate the variable prints a variable and returns a variable so pinpoint here on this slide is a login library that we will return to shortly it's basically print on steroids and it's included the reference to this library the dependency is also included automatically by sublime when i hit the hotkey so the thing about the compiler is that it contains a lot of a lot of methods that are called dozens of times hundreds of times during the life cycle of the compiler during the compilation life cycle and so those points in those points printing a variable might yield you a lot of a lot of input a lot of output and that output can be hard to analyze and what pinpoint library does is it addresses precisely this problem of analyzing the output as you can see every line of the output is assigned an id and what you can do with pinpoint is you can basically predicate your log on the ids of certain log statements so it means that you can have if you if you need to you can have another block output stream and you can write to that stream only once a previous stream has currently written an an entry with id say 7 or 4 or 5 or whatever id is relevant to you i'm not going to go into details of how exactly that works because we only have five minutes left so i'm going to skip the details of how exactly is this conditional login done but the idea is that it's it's necessary to filter out some output from the hot spots of the code base or methods of the code base what i'm going to do instead is demonstrate you how how the technique works in practice this instrumentation works in practice by a small videos that i have pre-recorded please let me know if you could if you cannot hear the sound let us see how to use dirty debug techniques to figure the cause of an issue in the bat scala file we have the issue in question the issue is that foo and bar compute essentially the same type yet bar fails to compile it fails to compile because the compiler thinks type parameter a appears in a contravariant position while being declared covariant let us see why this issue happens to see where spurious error happens let us recompile with white debug error flag for every error encountered this flag will force the compiler to print a stack trace pointing to where the error occurred let us now save the trace for further reference now let us go to the error site occurrence the very last couple of frames typically define error reporting logic and aren't very interesting so let us jump to the first frame that appears to be an actual program's business logic we can see that the error occurs because variance conforms check failed let us now get more context as to where this code is executed from to do so let us go a bit shallower through the error trace to obtain the whole picture one of the frames that may help us establish the context is the post typer frame post diaper is one of the phases of the compiler and having it in the stack trace indicates that the error occurred in that phase this is a valuable piece of information to help us orient in the code base and establish context let us see what is happening there apparently post-typer tries to run some checks on a lambda which exactly lambda let us find out by debugging the variables involved upon inspection of the arguments we understand that the cheque is run twice and the lambdas in question are the types that appear in our issue let us now see what exactly checks are being run by this function as you can see the command to that function indicates that its task is to check variances of a diplomata upon further inspection check lambda is where the error actually happens now let us investigate what in the code triggered the error the compiler errors because apparently variance confirms returned false there is a variance computed by the compiler and a variance derived from the type parameter sign and the two did not match let us test this theory by debugging the variables involved indeed for the type parameter b the variance is much but not for the type parameter a in case of a the compiler thinks variance should be -1 that is contravariant whereas the declared variance is 1 that is covariant let us figure out why the compiler thinks the variance should be controversial well i guess this demonstrates setup enough we have just one minute remaining but i mean the rest of the video is more of the same essentially there is a good showcase of how things are done um so yes that's um that's how my setup works in practice and you if you if you're interested in trying it out with sublime text you're welcome to refer to the links that will be shared in the slides the slides will be shared by the organizators i believe in conclusion i would like to make two points so it's um if something repeats in your workflow regular regularly and it makes a lot of sense to automate it and standardize it and you're going to benefit a lot from that and also it's important to know how to script your code editor the point of the stock is not to promote sublime text editor or my setup but it's the point is to promote the concepts and to promote hacking on your own editor to maximize efficiencies for your particular work so that's what i wanted to share with you thank you for your attention and if you have any questions you're welcome to ask [Music] you