SBTB 2023: Jeremy Smith, Aging: Evolving Software, Tools, and Ourselves.
Recording: SBTB 2023: Jeremy Smith, Aging: Evolving Software, Tools, and Ourselves.
[Music] yeah um so let me introduce myself first real quick uh I'm I'm Jeremy I work on the machine learning platform team at Netflix but this is my own talk I'm not here on behalf of Netflix um you can link me in I'm gentleman and a Scala on LinkedIn I'm Jeremy R Smith on GitHub and I have Twitter at the same name I don't really use it though other than for complaining um so uh if I look a little bit familiar you might have seen me here back in 2019 I was one floor up talking about polynote which is a Scola first uh polyglot notebook tool that I made uh I'm not talking about polynote today this is going to be the conferences uh obligatory talk about technical debt instead which is a really exciting to topic I know so why am I giving this talk instead of talking like plugging some OSS project or talking about some school cool Scola thing um and the reason is it's just it's been on my mind a little bit uh I found it useful to organize my thoughts about it and so uh the goal here is to maybe encourage you to organize your thoughts about it uh and if uh things that I'm saying are obvious or obviously wrong then that's great it means that you are thinking about it uh and that's kind of the the point here um and kind of the motivation is most of us don't really think about what we are going to be doing you know 5 10 years down the road let alone what's what our code is going to be doing uh so you know I think just making a plan for that kind of like a a a you know fiveyear plan for your code like you would for yourself uh can be an interesting exercise but why why am I giving this talk specifically um like what qualifies me to talk about this kind of thing uh and the reason is uh well let me get a show of hands who's ever had to clean up somebody else's mess yeah and who who's ever had to clean up your own mess okay good A lot of people um but you know I noticed particularly in Silicon Valley um it can be common to kind of you know by the time your code starts causing problems you're at a different company uh so it's not really a problem um but you know it's kind of unusual I spent 10 years at a company I've been at Netflix now for almost seven years um so it's kind of long 10 years and I've uh been been around long enough to make mistakes and live to regret them at at multiple companies so um that's that's kind of where I'm coming from uh all right so so this is a super scientific plot that I borrowed from Robert Martin of you know a typical software projects uh productivity measurement over time and it starts off really great uh it's exciting we're whiteboarding we're architecting uh everybody's uh making amazing elegant abstractions and cranking out features right and left um but over time and the the time axis is not to scale here obviously over time it's it can start to dwindle like productivity just sort of starts to slow down features take longer to build um you know progress just doesn't happen as fast and one reason that happens is technical debt and I'm going to Define this as anything that should be changed but hasn't been changed yet uh I I coincidentally heard another talk recently about tech de and they used the the the phrase like the Delta between the current state and the ideal state of the code which I think is a lot more eloquent way of putting it but I'm going to stick with mine because uh that would be cheating to steal theirs um so uh what we usually think of when we think of technical debt is the self-inflicted technical debt and this is kind of the stuff that we uh inflict on ourselves and our colleagues or our uh successors potentially uh for the sake of deliverables right and this is things like to-dos will'll do we'll uh we'll do this thing later you know I know I need to do this but I'll do it later um I'm going to do a quick and dirty thing right now just to you know hit that deadline I'm going to do a hack or a workaround that I know is you know probably not the best way to do it but I don't really have time right now to figure out what the best way is and this is going to work so that's what I'm going to do and I'll come back and fix it later right uh how often does that happen uh we'll talk about that um but then there's there's Tech debt that's what I would just call age related Tech debt and this comes largely from things outside the project uh and this is things like concept drift right so when uh you know maybe the software has to do something slightly different now than it did before uh or new features have come in into into the into uh consideration and and that makes it so our abstractions are leaky now and we need to go re-evaluate what they actually should be um so things like that um obsolescence maybe there's now there we implemented a bunch of algorithms but those are old hat now there's brand new fancy way of doing doing those same things and and we need to go uh Implement those new ways of doing it um and dependencies are a big one right we we depend on uh Library iies on on languages platforms uh and when those things change we need to do migrations we need to update our dependencies we need to move onto new platforms um you know we need to change from python 2 to Python 3 or Scola 2 to Scola 3 uh and putting those things off because they're slow and expensive and and not a lot of fun uh is a form of tech debt and and so you might ask yourself what what constitutes finished software and uh I'd argue like there is almost no such thing really as finished software because all software accumulates that Tech debt just by existing um just by the passage of time right existing through the passage of time a project is going to keep accumulating Tech debt uh even with nobody touching it right let's say we call it done and we walk away uh you know years later that project's got Tech debt if anybody goes to change it they're going to have a lot to deal with right um and and this comes from security vulnerabilities you know maybe you introduced one in your code or maybe one of your dependencies or your platform or your framework has a security vulnerability and then when they fix that you've got some tech debt because you've got to migrate um incompatibility so uh uh as libraries evolve uh your dependencies might start conflicting with each other you know we call this dependency hell um if you don't migrate your language you won't be able to update your platform because the new OS doesn't have the old uh python or whatever um so these things accumulate uh and worst of all hindsight accumulates so software is made by people uh people hopefully grow and change over time so uh and the world changes around you as well so things that seemed like a good idea at the time can become huge mistakes takes retroactively right and that's a form of tech as well um and these things tend to compound with each other multiplic multiplicatively um just like debt so what's the cost of keeping Tech debt around it's like what's the interest you pay to to use a weak the weak debt analogy um there's some direct costs of this right there's a loss productivity like we saw in that scientific plot at the beginning um Tech debt tends to slow you down in various ways uh there's lost agility meaning that uh the the needs of the software changes or the market changes your users change and you can't adapt quickly to that uh because of tech debt slowing you down um and of course there's costly failures that can come out of failing to maintain uh the software this is just a couple of examples that might look familiar of uh you know cases where old outdated Tech debt ridden software that nobody had time to update caused a lot of financial you know economic cost there's also indirect costs though of tech debt uh software doesn't exist in a vacuum uh all software has users and a lot of software is used by other software in turn so if you're working on libraries or Frameworks or platforms um your software has has your software project has other software projects that depend on it um and this creates Network effects when it comes to Tech dead and I think the the indirect costs are the more interesting part to me because I think it's the part that we don't think about as much um and even if we think about it we might not think about the scale uh of it and we'll talk more about that later um but the point here is a Project's Tech Deb is also born by all the other projects that depend on that project right even transitively so if you have Tech debt in your project anybody that's using your software as a dependency also has some tech debt uh from that in their own project um yeah so Tech debt kind of begets Downstream Tech debt and that can come from like code problems with your code uh maybe bugs that are that are uh introduced by by you know failing to maintain it properly um leaky abstractions uh incomplete features that leave people having to do workarounds because your software doesn't do everything that it needs to do um uh award apis that leave people's code looking messy or things like dependency hell so paying it off there's uh a few strategies I'm going to talk about for dealing with tech debt uh and I'll do another show of hands here who has seen like a to-do or fix me or something like that in their code in recent memory in any code in their company okay great uh keep your hand up keep your hand up now keep your hand up if and only if you fixed it be honest okay all right I see a couple hands up that's good but a lot of them went down uh and I'm not I'm not here at judge I think that's uh ignore it is a very uh popular strategy for dealing with Tech de uh it's like Kick the Can down the road it's like an interest only loan for example um and uh you know that that can be tempting because uh it requires no work right now um and we're all very busy um so make payments uh is what I would say like incrementally chip away at the to-dos um we can migrate things that need M migrating we can you know be diligent like some of you who see it to do when you fix it right there that's great um but that can be tricky to do it can be tricky to find time for that um there's also like you could think of like larger lump sum payments maybe will'll take a week and everybody's going to work on te you know Tech debt related stuff for a week um that that that happens as well um but you know it can be it can be tough to uh to find time for any of that um and then there's uh the the rewrite right uh like declaring B bankruptcy sort of but with with less paperwork um and this is a popular one too because you know knowing what we know now this will be easy we can start from scratch uh Tech debt free Clean Slate Green Field uh it'll be fun it'll be easy um so that can be very attractive um so one so a couple reasons I think uh making payments uh can be a tricky proposition um and this is kind of like the treadmill but it it takes a huge amount of discipline to incorporate this into your daily practice which is probably the most effective way of dealing with it uh so if you have that discipline that's great um a lot of us you know struggle with that uh and there's there you know it's understandable um and Tech de can also accumulate faster than you can pay it off in with that strategy right uh uh you're not the only one on your team maybe your colleagues uh are adding workarounds and and hacks and things maybe uh your dependencies are evolving more quickly than you can keep up with um but it can accumulate too fast to be able to pay it off that way and it could be hard to justify those larger uh chunks right we're going to take two weeks to work on Tech debt um you know maybe you tell your manager we're going to spend two weeks of resources to keep things exactly the same that can be a tough sell um uh in at some companies uh and then on on the extreme opposite end of ignoring it there's the there's rewriting it so it can be really really tempting to say hey let's just start over uh because that that can be a lot of fun it's really tempting for engineers to to have a green field to have a clean slate um because we can recapture that feeling of being very productive right remember the graph at the beginning when we first start a project we're super productive and it feels great great um and we can do it right this time we could do it modern you know we'll make it we'll make we'll make it much better this time around uh but something to keep in mind is for a project that's been around for a while it's accumulated a lot of functionality presumably uh and we can we can we tend to underestimate how much functionality uh is there and how much we'll have to uh rewrite in order to to make this work so how much work and effort how much time is it going to take to reach feature parody in your rewrite um it can be easy to underestimate that uh and we can also fail to think about like what's going to stop Tech debt from creeping in this time you know uh it happened before what why isn't G why won't it happen this time uh even if we do get to you know reclaim that productivity bump from the Greenfield project uh how are we going to keep Tech that accumulation at Bay this time around and not not to mention while we're rewriting the the system what's going to happen in the meantime so we have an existing software that everybody's using we're telling them you know we're going to stop working on that uh it's not going to evolve anymore while we take you know however long months years to to rewrite the whole thing from scratch um adding no new functionality at the end of that so that can be a difficult cell for for your users um and we also can't assume on the other side of it maybe we can't assume that the existing thing is going to remain stagnant while we while we rewrite it so uh a lot of times uh the rewrite uh arose by some new feature that nobody wants to implement in the old system or it's too hard to do it because of all the tech debt um and so is that new feature just going to be on the table while we rewrite the whole thing or uh you know what's what's likely to happen is somebody has to go in and Implement that new feature anyway in the old system while we're doing the rewrite and now what motivation does anybody have to adopt the Rewritten thing afterwards um because that's a that's going to be a migration for them right uh so you know just uh you know a rewrite can be successful with a lot of discipline and a lot of talented Engineers um and a lot of mandate from management uh but make sure you have a plan for that make sure you have a plan for uh they're being a real realistic path to adoption of of the new system and motivation uh for users and look at it from from users perspective um and so uh here's kind of a uh one way to think about whe like whether or not a rewrite makes sense um you could think of it like we're the project is sailing to a destination and maybe we've drifted a little off course right uh so there's the place that we want to be there's the place that we are now now uh and if you think of these as sort of points in some ocean of of space then uh you know think about how far we've come uh I guess in this analogy you also have to imagine that you're able to teleport yourself back to the back to zero instantly but not teleport anywhere else so it's kind of a leaky analogy but you know if if the distance between the current state and the target state is less than the distance between zero and the target State um you know the target State being defined by some measurement you know hopefully it's a a business oriented measurement and not you know a developer satisfaction measurement um but you know you'll get different uh different ideas here based on what what criteria you use there um but that can be a useful way of thinking about it uh so what's the total cost of paying off Tech debt so there's a direct cost um which is you know just the engineering resources of whatever strategy we chose whether it's incrementally paying it off whether it's rewriting or ignoring U there's that direct cost uh and there's like an opportunity cost of doing that instead of doing something else you know you could be uh devoting your resources to adding new features to the existing thing instead of paying off Tech debt so what aren't you doing in order to to work on teet but there's also like an indirect cost which is a cost that's like not born directly by your organization uh and I think that's why we don't think about the indirect cost as much um but what happens is something like this this is a famous uh XKCD comic so uh those persons fixed some tech debt in their software uh and that's broken somebody else Downstream who has been using it in a horrifying way uh which happens frequently right so and it's true every change like anytime you change software that has Downstream users uh it's G to break somebody uh when a project pays off Tech debt it creates tech tech debt in Downstream projects right um and like we were talking about before this can have Network effects so if a project updates its apis uh like say your zo and uh you're thinking oh we you know instead of naming this method for each underscore we should have named it for each discard um and maybe that's true you you should have done that but uh if you change it now a lot of people have to migrate to that new thing right uh so that's Tech debt for them so uh you know and if uh if you update your dependencies in your project that will create Tech debt for you uh or for people Downstream of you um based on that and so the the network effects kind of cause a lose lose here right we were saying before if you don't pay off Tech debt it creates Tech debt Downstream uh and if you do pay off Tech debt it creates Tech that Downstream um and I think we we sort of we tend to underestimate the scale of network effects um and this is this is actually a illustration of a graph in hyperbolic space but I like the exponential uh nature of it um like you you'll see somebody making a decision based on you know what it costs us directly like what what what's the development speed for example of making this decision um but if you think about the work that it's going to create Downstream and downstream of that and downstream of that uh the total cost can can be a lot more than than what's obvious just by thinking about you know the local cost uh so I guess the point is to weigh the total economic benefit of the the changes you're making or the the tech debt you're paying off versus the total cost of what that's going to be uh taking all the you know all the network effects into account um so so if we're going to get Tech debt no matter what uh paying off Tech debt is going to create Tech debt not paying off Tech debt is going to create Tech debt Downstream um does that mean everything's hopeless uh I think it's not hopeless uh because even if you can't eliminate this problem you you can minimize it uh by being mindful uh of the tech debt that you're going to create so some so some points here are compatibility is key right if you support your apis uh if you have stable apis or stable abis and this is like you know in in jvm land this is binary compatibility uh then that can truncate the network effects of of uh of this paying off Tech debt of creating Tech de um so I like a shout out to cats here is I want to a shout out to cats they they've had forward uh I mean they've had backward binary compatibility for years in cats uh that's not easy to do even through API changes they've they've managed to keep binary compatibility and uh that's why cats doesn't really contribute to dependency hell as much as other libraries do um it's because of that that commitment and that can be huge it's not a Magic Bullet uh backward compatibility is not the whole story uh for example if you have uh Frameworks or platforms that distribute some you know binaries with it like say spark for example uh you know if they're using an old version and there's no forward compatibility then people who are using new versions can can have uh uh conflicts as well so it's not a Magic Bullet but it's a it makes a huge huge difference uh and Scola 3 has this target name annotation uh this is one tool for intentionally keeping binary compatibility um it's not even the intended purpose of that annotation but it could be used for that I would love to see a lot more uh features you know in the Lang in languages uh that are dedicated to being able to evolve while keeping compatibility um I think that would make a a huge difference like if I could say this method is no longer usable from Source but it is going to be there in the binary uh just so you know people who who are using an old version of the ABI can can uh can still upgrade or use a a new version like that would be huge uh and Sac sacrifice things at the altar of stability so the the jvm platform itself is a great example here uh it's evolved over the years um but it's evolved like very slowly and intentionally and thoughtfully um and you can see the thought that they put into it you can take code that you compiled you can take Java that was compiled in 1997 and run it on today's uh jvm and that's uh you know not easy to pull that kind of thing off while still uh making progress on your platform but uh they sacrifice you know speed at the alter of stability and that's one reason why it's such a successful platform I think um all right I want to say one thing about migration tools so uh schol fix is a uh a tool for automatic rewriting and that's awesome it's really great can be very helpful uh when you have to do uh uh migrations but when you are evolving your software if you don't want to create Tech debt Downstream or you want to minimize Tech debt Downstream pretend scholar fix doesn't exist and the reason I say this is uh don't like don't make assumptions that your users have scol ofix or will be able to use it uh or it will work perfectly for them and I see a lot of uh a lot of sentiment lately like reite tools will take care of it you know and this becomes sort of a a magic Mantra sort of uh akin to Oh The Hot Spot Will optimize that away you know and people like repeat that to themselves without really thinking about whether it's true and a lot of times it's not true um and it can be like changes that are made on the basis of it's okay because rewrite tools will fix it th those can be dangerous because it's an excuse to avoid intentionality in the changes that you make um and that kind of worries me like I've seen this especially with uh with regard to the evolution of the scholar language itself there's a lot of sentiment of well we can just make you know large sweeping changes willy-nilly because rewrite tools will will solve it for everybody and I think uh it's we're going to reach a point where that it becomes clear that that's not always true um and that's that worries me because I really like Scala I want it to succeed and I think uh stability is is pretty essential to to the success of a of an ecosystem uh and one example I want to give is is scholar test if you work on scholar test I really appreciate you uh this is not I'm not trying to pick on scholar test but it's a recent uh example that sticks out to me which is they so Scola test it's a a testing tool for for Scala most of you if you've used Scala you've probably encountered it um and they you know they had some tech debt they wanted to introduce the ability for tests to return typed values or use typed values um and they hadn't had that before so how do you move from you know the where we are now to to having that I assume that there was you know two choices that they could have made I wasn't there so I'm not sure but uh it seems likely that they were thinking of well we could start a whole new hierarchy that has the types in there and then people could move to that as they as they please um or we could stick the type thing in the hierarchy and make the current thing you know just a special case of that uh and that required them to re like anybody who didn't want to opt into the types uh would have to change their like all of their tests uh you know to to do that migration and I I suspect that they chose the ladder uh on the back of scholar fix because they did offer scholar fixes uh to help with that uh but those scholar fixes don't work perfectly uh and uh this is just anecdotal but I know from from my personal experience like I spent at least a week on migrating tests to the new scholar test and that's not you know something that felt particularly productive um so that like that if they had thought about the network effects a little more if they had you know thought about what do we do in the absence of scolf fix um I think they might have made that decision differently um so yeah definitely use it and definitely provide schola fixes but uh don't make choices based on the existence of schola fix uh I guess is what I'm trying to say there all right so just to make your regret sitting through the whole talk here's the takeaway in one slide um I'm saying Tech debt accumulates even from the outside uh whether you you create it or not uh ignoring it creates Tech de Tech debt Downstream fixing it creates Tech debt Downstream uh you can minimize this by being mindful about the changes you make uh and don't underestimate the scale of network effects uh yeah and that's that's it thanks so much for for coming