scale.bythebay.io: Leif Wickland, Defusing the configuration time bomb with PureConfig and Refined
you hello I'm Leif Wicklund I'm gonna get started I have the pleasure to live in Bozeman Montana way up in the frozen tundra I get to fun things like go back country skiing and hiking and that kind of thing I worked for the Rubicon project you and I both like free things on the internet and as it turns out most of us get those things paid for by ads and most of that ad money runs through Google and Facebook these days which takes a bigger cut than you might want so like Rubicon is one of the other folks that like helps publishers and the people that make all those nice free things that you like get those things with a bigger cut of the money which is to say I work in AD tech and I get really big data really fast so under late oh I'm sorry I'll try to be even louder so what's a program I hope your programs are algorithms and when I was you know learning about CS there was an analogy they used for what an algorithm was and at the time do you guys have any idea what so they they said it was like a recipe like you started with some things you followed some steps you got somewhere I'm like all right when I'm following a recipe what's a common mistake I make well I get super excited and I forget to ensure that I have everything I need before I get started so I'm like I'm gonna make some awesome pesto I'm gonna get so excited about my garlic and my basil and and there's got to be some other stuff but like Oh garlic man I love me some garlic it says I need to close and they're gonna be cloves that look like this not those wimpy little cloves right I'm gonna toss that and my pine nuts into my food processor and so jazzed and and I got my awesome green sludge at the end right and the problem is I got so excited I forgot to make sure I had my pecorino and instead I end up with a sad green sludge when I was practicing this talk my friend was like when you forget the pecorino it's just vegan and I'm like okay right so I'm not gonna make the joke there that's obvious about sad green sludge is vegan because he did it for me but if I start with the ingredients and I'm and make sure I've got all of them and I follow the steps I end up with good things likewise in my code if I ensure I have all the input I need and it's you know pretty solid before I star and then I run the code I probably end up with some output that's pretty good so it's the input to our applications data what kind of data okay configuration somebody read the title of the talk that's excellent but besides that what might we have we might have input from our users or we might have some sort of data store that we like write data to occasionally and like read it back and maybe some things change in between but also of course we've got configuration which is you know obviously from the title why we're here so I think our industry has gotten better it's sanitizing our user inputs and our data stores like if you use Rob's amazing Dubey library for your database work it's really hard to find yourself in a place where you get an SQL injection error you know I mean not the whole industry has gotten better but in general like if you're using good tools it's a lot easier but I don't think that's true for configuration most of our configuration just looks like any of the string all the way down like instead of turtles it's poorly described functions all the way down so I'm not the only one saying this like there's this nice paper that came out where they investigated what sorts of things were causing people problems with their configure or with their systems so what these guys their thesis is the configuration errors have become one of the major causes of failures in large-scale cloud systems and they had a bunch of good examples so they had a app engine going down for two and a half hours in February of 2010 they had AWS and ec2 going down for days they had as you're going down for a few hours back in 2012 so like let's dive into one of these and see what went wrong back in 2010 App Engine went down because the power went out to a particular data center and then when the backup power came up it only went to part of the data center it's the nice people are operating this system had to configure things to avoid those boxes that didn't come back up but when they did that they made a mistake in the manual editing of the configuration and of course nothing worked after that and it stayed down for a while longer while everybody scramble to figure out what was wrong so my own life at work we a homegrown system that manages data retention and mirroring we needed to make a change to the retention so you know it created a G R I went through all those happy things we changed the retention time since P R we said yeah looks good stamped it with the the squirrel of shipit and got merged I went to Jackson Hole for the weekend to do some backcountry skiing syntaxerror hits the config file the app starts failing this is hours later after the things been murders because this runs like once a day right nobody notices that because it doesn't have good reporting on it sadly the pipeline stops like two days later then we noticed that our reporting pipeline has stopped and nobody's happy and engineer start getting called on the weekends so the paper that I referenced before has this quote it's hard to completely avoid errors so a more practical approach just to detect errors as early as possible in your configuration and minimize their damage so why isn't it enough just to start up the application and see if it failed well kind of like in my case with the retention and mirroring application it did start up and when it started up nobody was there to notice it because it started up quite a bit later but I have other jobs right that I run them and they run for hours and then they need some piece of configuration and if it's busted then I've wasted hours of cluster time and I didn't find out so another example of that came from the same paper somebody reconfigured squid that the proxy to change the way that logs are rotated squid doesn't validate that configuration on startup so then days later when it went to rotate the logs the error is exposed and it caused squid to go into a busy loop just using all the CPU on the Box the poor system administrator gets called early in the morning is like what's going wrong I haven't changed anything in days and it takes him days to figure out why things are wrong right this is an example of our kind of configuration errors that that hang around and hurt us so the authors of this paper call this a latent configuration error I'm gonna call it the time bomb that's ticking so so how how can we figure out how bad the prob these are well conveniently the research did this for us they looked at a bunch of open-source projects that I think a lot of us depend on and over on that right column you can see like in HDFS they looked at 45 config parameters from there and of those 40 sorry 44 of those 44 41 had no validation at startup and three of them had some validation but not enough like if we come down and look at MySQL they're in quite a bit better shape only 12 of the parameters of the 43 of the studies for that had some or no validation right so you know but all of these things have some amount of parameters that aren't validated and then of those parameters that weren't in some way validated they said which aren't used during installation and so we're still finding a bunch that aren't used and these are the ones that have the potential become late in configuration errors so the short answer on this paper is yes latent configuration errors are a prevalent thing so does it matter well they claim it does they say 75% of the high severity or errors that they identified or a result of this kind of problem so how can we fight it what can we do to like prevent our systems from going down in this way when I was trying to pitch my talk I was talking about it on Twitter and somebody replied to me and said just be a decent human being use continuous integration and continuous deployment and QA and and you'll make this problem go away I don't think that's true I don't think based on the examples that we've seen from the paper and from my experience that that's going to be sufficient because it's these sorts of errors that come up days later when we have long-running stuff so what could we do to combat these we could validate our configuration on startup right we aren't gonna just have values that we keep in a string to any map indefinitely until we need them days later we're gonna look at them and start up we're not going to just say like there's a value present but we're going to say it's usable in some way and then we're going to make it possible for people who maybe are outside of the development chain who have to make configuration changes to validate the configures right before they start the apps maybe okay well this paper published proprietary software not published they pushed proprietary software to do this they didn't publish it I'm gonna try to do better and show you some open-source tools that I think can help you do these things what I'm going to show you is real code if you don't want to try to read code off of a screen you can go to this address I've got some example projects there that kind of walk through all these things so here's a motivating example we want to create an awesome client which I don't know you can fill it in however you want it's a thing that's gonna go to the Internet it's got some API key to get permission and when it gets whatever awesomeness from the Internet it's gonna write it out to some path locally and if it can't find it within timeout it's going to give up like this seems like a plausible thing we might want from the internet and we probably wanted to be configurable right so I'm gonna write me a config file and like usually our config files are gonna be gross like this we're gonna have some timeout in here that's dimensionless and like we're not crashing spaceships into foreign planets we're you know we're like just trying to configure an awesome client here why would we need units right and we'll just keep going along like this and then we've probably got some code that creates a properties loader and like reaches in and grab some strings and then converts the strings to the appropriate types like are too long hanging out over there or just creating a bearer URL over here alright and this I hope it looks similar or maybe I don't hope I think I've constructed a reasonable facsimile of a lot of code that we've seen before at least I've seen a lot of code that looks like this in the time I've worked there there's a lot of things about this that are suboptimal right we're mixing together application logic and we're mixing together our configuration logic we're not using that config logic until we actually need to create the thing so we don't know if that's right and we have to keep repeating our string conversions right so like on this previous page I've got a spot I'm converting my numbers to a long I probably have more than one number right but I'm kind of having to do these same conversions over and over again and our data types while giving us a little bit of structure they're not giving us much for free either and I think we can do better than that if there's an error in our config files you get an error that look like this exception right if I put sad in for my time out well then I get an exception that doesn't tell me where and my config file things are broke and it just tells me where in the code things went wrong which isn't nearly so helpful when I'm awakened at 3:00 in the morning I'd like to know where and the config file things are busted so that I can go fix it right away and we don't get an error until we actually construct an awesome client okay we can't actually like find out that we have a broken config like I was saying without actually starting up that application and running it to the point that we need our client okay so how do we solve problems in computer science like the general answer is of course we add another layer of indirection so the layer of indirection here is let's add a config object that we read our values into and that we can ensure is well-formed before we get on into the application and we need our awesome client right so I've now got a vanilla config that I'm gonna read these values into and it's nearly identical to what it was before except instead of reading the values directly into the awesome client I'm now going to read them into my vanilla config so I've made a little bit of progress right now I can at least address that thing I was complaining about where I can't verify ahead of time that my configuration is good I can now do that right I could start up my app create a vanilla config and yep okay so those things look roughly right I at least know I have the right config keys there that sort of thing but there are still some shortcomings to this so I have to like write manual conversions between every field and the you know bits in there so I've got my kebab case in my config class proud config file probably and I have to then map it over to the field named whatever that's subtly different I have to explicitly convert everything from strings to proper type still and my configures are still kind of the gross exceptions to instead of pointing at the config file well I've had good luck with a library called pure config and let me show you how it works so pure config magically maps the field names to the Figg names so you're never having to do this sort of thing again that's being generated for us it converts from the strings that are in the config file over to intz and other nice things based on the types of the fields that are going to receive those values and it builds on top of typesafe config so we can use the Hakan format I hope you guys have heard about Hoch on I think it's pretty handy it's sort of a superset of JSON and the properties syntax pop quiz how do you write a comment in a JSON file oh I mean you write a string of course right that's labeled comment which is terrible and hard to see because there's no like highlighting in the file that shows you this is a comment and different from everything else so anyway um yes I agree with you you can't write a comment in JSON it's awful it's got some other nice features where we can merge in properties from multiple files and from the environment and from - new properties pass to our java process and it gives us really nice error messages so if I pass in an invalid duration in my configuration it'll tell me which file things are broken in so what does this look like I'm gonna use a new configuration file here that's in Hoch on format it looks quite a bit like the other one but we're gonna use colons here just so you can tell that I've made it different and sort of vaguely json e and my time out instead of being 15,000 whatever's is now 15 seconds because I like units all right so I've got my new config file up here on top and I've got my shiny config that I'm gonna read things into instead of a long like we had for the vanilla timeout I've got a finite duration because I like to use types to represent my data in meaningful ways and and then to load it I'm gonna have this line and pure config using the mighty power of shapeless is going to generate all the magic to load the data in there and that's the end of the story so this seems good but I did lie to you there's no part of this that's magic it's verified at compile time it's well-documented it's configurable it's extensible it's got a bunch of built-in support for common libraries I say that it's verified at compile time I'm not trying to say that we like magically reach out and grab all your config files we do in fact know that we have converters present so that if you have say a URL I in fact know that I know how to turn a string into a URL or if I want to have a file I can turn the string into a file those things are nice like if I'm converting say XML via some sort of reflection horror into things at runtime I may not have those and I'll get an exception which seems less good the documentation is pretty intense I don't really know how to show you that the documentation is intense in the sense of we've done a lot of work on it and there's a lot there and lots of examples besides show you this beautiful table of contents out of generated thank you it's configurable so you can give it lots of different ways to say this is where my configuration is coming from I have more details on that if you're later interested later the mapping of kabab case to camel case that's optional there are other ways to do that as well the conversion to turn a string into the nicely type thing is entirely configurable say you got somebody that was very cute and thought they would use values other than true and false and their config files right so it is my service and they able to yes in fact it is quite lit or no this is quite a basic service which is not enabled well I could write code like this that would then be used by pure config to say these are the strings that could map to true and false and for other things I'll get a nice error message saying I don't know how to do that conversion I say it's extensible so I do some work with spark when I do I need Hadoop's path to be able to tell spark where stuff is peer config doesn't have support for Hadoop because there are lots of different flavors of Hadoop and I have a particularly nasty one so out of the box when I put a path into my config file and ask pure config to load it the compile time fails like exactly okay I want I don't want to get to run time and find out that there's not support for that so that seems good so then I have to add code like this where I tell it hey when you're reading a config and you find a string that wants to become a path well try to make a new path out of it and and I'm done right now I've added support for this other library on the topic of libraries we've got lots of them that are already supported with modules in appear config cats Hakka Java time I worked on the Jota one because I had Jota instead of Java eight squads is a really cool library if you do anything with dimensions it it's if you do anything with like sort of scientific dimensional sort of analysis squats is a cool thing and and so when married up with here config you can write in your config file that I need 40 3 angstroms here or you know 46 light-years there it's just kind of a cool library so coming back around what can we do to make our configuration even better well I know this API key probably shouldn't be an empty string because usually at least on the internet if I have something that wants an API key it's not going to be happy with an empty string and I'll bet you if I set my timeout to a negative number nothing good is going to happen so I don't have a way in pure config or in Scala to express those kinds of constraints but actually I do there's this library called refined that lets us do type level refinement of the values and before I heard of refined I didn't know anything about this so when they just say refinement what they're talking about is we are data types like int it is a way of like it's more about describing the data storage right it's saying here are 32 bits go do particular thing with them what you want but that doesn't usually encode the intent or the logic of the range that value can actually have refined allows you to dial that down to a specific range so at compile time you can tote around the fact that this value is not just an int but it's an int that's bounded to a particular range of possible values and then at runtime we it adds validation to ensure that the values that appear actually are that and so let me try to make this little more concrete I said my API key needs to be non-empty I can say it's a string that's refined to be non-empty and in fact at runtime it'll ensure that that's true or I can say my timeout has to be a finite duration which is refined to be positive and again it's true so the syntax here hopefully isn't terribly confusing refined is just a type that's sitting in in fixed position it's a type that takes two type arguments it so it that's really refined to string comma non-empty but if you write it like this I think it's a little bit easier to see that so with the module that allows pure config and refined to marry up I can write this and if I put a negative value in my config for the time out then I get an error message out of pure config that looks something like this I added some white space so it fits on the slide a little bit better but it's gonna nicely tell me that we can't convert negative 15 seconds to a finite duration that's positive because 15 seconds isn't greater than zero negative 15 seconds isn't greater than zero we can do a lot crazier stuff with her find so inside the brackets here you can see I've got W 1024 65535 so what I'm trying to do is define a data type that describes a port number right and port numbers can be from 0 to 2 to the 16th minus 1 but you know I I'm not gonna run my services route because I'm not that kind of noob right so I'm gonna keep it outside of the system range up where 1024 is and so now I know that when I get a value out of my config file I've got a safe value that I can bind to well-refined already has this defined they've got a thing called a user port number and they were smart enough to read the TCP spec not like me and they know that the upper bound that I'm supposed to use is actually 49 151 but whatever right like these guys know what they're doing I don't that's cool so there are other cool things I can do like for my API key it's a grid so if it's a grid why don't I use a rig X and ensure instead of it just being non empty that it's exactly looks like a grid so I'm gonna have a ray gags that can take uppercase and lowercase letters numbers dashes and it's got to be exactly 36 characters long and so this will ensure that at runtime for me these are magical things in refined that I repeat as incantations and magic happens I don't know exactly it works when I use them because again I don't actually need these most of the time because smart people have come along and built nicely named versions of these things they have a thing and refined called UUID that does this for me and does it even better where it ensures that AG wit has the typical pattern that you expect to see so I can do effect 'fl things with my validators and refine so these this runs at the construction time of when I'm creating a value that is refined in some way and when it does it gets some integer in and it then opens up a socket and then it immediately closes it just to verify that the socket is available so yeah it's kind of weird like we're on the bound of like doing something that looks like it should be pure but I can also do something effect full here or say if you've written any spark you know that you can go off a write a long job it'll compute for you know tens of cluster hours go to write to disk and then you gave it a path that's already I've been written to you before and it will fail and all your delicious bytes are lost because there's nowhere to put them so I could write a validator for refined that let me say not only do I have a path but it's also a path that can be written to and if I'd been a little bit more ambitious I could have said like it's not a directory that's there but I can write to its parent but that doesn't fit on a slide as well so just play along here we're gonna say that it's a path that exists and is a directory and I can write to so I'm gonna get some P in I'm gonna ensure that I can write to it I'm gonna say that it's also a directory and if it's not then I'll write a nice error message saying that it's not a writable directory it seems kind of cool so can we take this too far for sure have we taken it too far I'm not sure like one of the things I said it'd be nice to be able to do up front is when somebody makes a change to the config I'll have a app sitting around that they can run that will read that config in and tell them if things are broken if I start adding these affect full checks then they have to run it in the exact same environment like if I'm just gonna run them on my jenkins box as part of CI or something like that well you know Jenkins is probably ready bound to 8080 so if I'm ensuring that 8080 is available like it's gonna give me some false negatives saying that it can't bind like you know that I'm not sure like there are some interesting capabilities here I have not had a chance to explore them but I can certainly see pitfalls in the neighborhood this is just kind of saying if we've gotten to this place where we're using pure config and refined so that we can load up our configs and we've really well described what our configuration looks like then writing an application that somebody can check ahead of time if the config is good starts to be about as simple as this right I mean we've just got a main that tries to load our configure throws and prints it out and we can write you know a shell script that contains something like this and we're done right we've made it easy when I mentioned this to one of the ops guys it was like oh that's awesome we'll just run that first thing in the container and like have a fairly exhaustive check on whether things are likely to work and so our latent configure is if we've done this right hopefully have been snuffed out to the best of our ability so we verified our config at startup we're verifying our config values are good we can run a check outside of our application to make sure that they're good what else can we do so we've got some tips some war stories if you will from using pure config one of them is you probably want to try to help the compiler peer configs built on top of shapeless it does lots and lots of derivation magic so it's really not the fastest compiling thing Adrian's claim is about you know very fast compiling we will do our everything they can to prevent that using peer config so I recommend that the way you combat that is you keep your configuration classes they're going to be loaded by pure config in a separate file or a separate namespace just to try to keep them isolated so they don't have to be compiled so often as you're changing things so I hope that makes sense the compiler is not going to help you terribly much at least my experience with other shapeless oriented things and certainly with pure config is oftentimes the compiler error messages are not only inscrutable but effectively misleading so if you have an error if you if I added say that Hadoop path to my config class then instead of telling me I can't find a config reader for the Hadoop path it's gonna tell me in my config class doesn't have a configurator which isn't gonna make any sense to me until like I kind of learn that that's the problem so instead what I've learned to do is sort of work incrementally and start with like a very simple class and build up from there and have some unit tests that run on it that seems a little bit better I'm sorry unfortunately the compiler is lying to us in a lot of cases here so where the config files loaded from people coming to peer configure often confused by this kind of the default that you'll see in a lot of examples from pure config is the type safe set dial the type safe config style so what that does is it loads the - deep properties and the environment VARs with the highest precedence and then it looks for a file called application Kampf like all over your class path and then finally it loads a resource comm file and it mashes all that together and it's entirely happy to ignore missing files I don't like that very much because I want to know when my files are missing I kind of paranoid like that there's another thing that pure config has added this load config from files that you give explicit paths to and it tells you and things are missing that's another way to go it doesn't slurp in environment stuff you can also roll your own so I have a colleague at work who's coming from like a spring boot background I took pity on him he wanted to have a different way of loading things that reminded him of home which is to say spring boot so he wanted like this interesting cascade of different environment files and stuff like that we could use the config factory from typesafe config til I give him all the ordering he wanted build up typesafe configs config object and then pass that to pure config slew config and he could get the match loading at that point so he could get all the precedents rules that he wanted it's it's a really flexible thing it's just something you kind of have to actually think about but the examples make it look like there's sort of one route down that pier config works on how to map the fields I said earlier that it does kabab case to camel case I'd rather have camel case to camel case that is my config file look exactly like the names and my case classes so to get that I put something like this in scope at the place that I call load config and it works out there are other ways to do it too there are other naming conventions that are supported if you have one name that for whatever reason has to be different in your config file then the the case class field that you're loading it into so in my case I've got in my config the Batman and I want that to become Bruce Wayne in my case class then I can write a product hint is what it's called another implicit that does that mapping so I set up my overrides there in that map saying Bruce Wayne will become Batman my config field mapping is those and other than that it just falls back on the default camel case to Bob case mapping one of the things I think is really nice about this is we're just sort of building on the fact that things and Scala are functions so like a config field mapping is a function a map in Scala is a function with default takes a function so you know we're bolting together a bunch of functions and we end up with a function that map's a string to a string that's the material I've got do you guys have questions you support business for good nesting yes the question was does pure configs support nesting classes within classes yeah I didn't try to fit that onto a slide but absolutely the question is are what other art exists in the space and could I provide an intelligent comparison Cirrus I think something like that and ficus I think are other examples in the space I don't particularly know them or or how they compare I'm sorry I think the question is roughly it what part of the validation can happen at compile time and what part can happen at runtime and sorry I there's only so much you can do a compile time the part you can do is like ensure that for every type in your class that's going to receive the configuration we have a converter that exists we cannot prove at compile time that you have good values in your configuration so no like you can absolutely put garbage in your config that will blow up at runtime and hopefully when it does we give you an error message that helps you understand why it went wrong if please use me already in sin so we can think can I wrap the rapid in purification employee application at the framework already gives me a config that it's read from all those different sources can you wrap it in this so I can apply all those checks or I have to go deep into the framework to change the way the question is I have a configuration file from play can I use pure config in some way to help me validate that it's good I can't think of how you would because what pure config is doing is mapping the sort of ast of typesafe config into well typed objects and it sounds like what play gives you is something that's already been mapped over so I don't know how you could apply those I mean you might be able to use that I don't know I'm sorry thank you [Applause]