scala.bythebay.io: Frank Austin Nothaft, Processing 100's of TB of Genomic Data With ADAM And Toil
I you in interpolating and doubling strings like a boss I can't interplant generalize interpolation like a boss yet so hello everyone thank you for coming despite me not being on the schedule I I think this is this is payback from what I did at scala world where we had about 25 speakers back in September and I didn't tell them when they were speaking until 4am of the first day of the conference so Alexia has paid me back I mean he wasn't one of the speakers but nevertheless he's he's applied the Karma and he told me this morning i was speaking so i have this this talk about interpreting strings like like a boss and I'm going to take you through a number of different levels of working with a string interpolation now I'm sure everybody is very familiar with the most basic string terp later somebody some people have probably used more advanced ones and by the way would you just give me a litigation whether you're you consider yourself sort of beginner level scala they don't hear a beginner some miles Paul okay does anyone consider themselves sort of advanced so you're all liars it is I'm sure your yeah we'll see how we go the first the first third of the talk will be very straightforward very much familiar Scarlett it does take a turn to the more advanced for the second third thirds but hopefully you'll sort of follow along if you don't if you don't understand what I'm talking about I'll share the slides later and hopefully you can just go back and sort of make sense of it but we're going to start at the beginning the value level and with a very simple example which I'm sure you've seen you've seen things like this before we have I have a laser pointer we have a boat looks like a little string here prefixed with the letter S and we substitute this string here boss into the position here indicates by the dollar there's nothing difficult yet we can you are slightly more advanced and we can actually use use braces to put an entire expression in here so interpolating now clean one tell me what this says when you actually run the interpolation interpolating java dot Langille strings like a boss maybe we can do better than that later we'll see but I mean you can see any any scholar expression can be substituted in that position there so this gives us quite a bit of flexibility s is just an ordinary it's one of a potentially many string interpolators we can give them any name we like any any identifier can go there and i'll show you how you can choose them later now what you saw there gets translated in the compiler to may be more familiar code code that covert existed before these were introduced in Scala two point nine ish so that that on the previous slide will be transformed during the passing phase of the Scala compiler it will transform it into this we have a new string context object created this service the case class with parameters which are always string literals and they're the bits of the string that are known at compile time they're they're the constant bits we exclude during the creation of the string context we exclude all the variable parts and then given that given that object we've just created there we apply the method s to it now s is the basically the text the identifier whatever that happens to be that appears prefixing the string literal so here it is s and then we apply parameters to this which are the variable parts so a key distinction to make is there are the the fixed bits here here and there is an empty string at the end don't forget that it will always be there but it might not be empty but in this case it is empty but there'll always be a a string constant there and then there's the the variable parts which gets slaughtered in now it's important point that this happens in the parser so keep that in mind for later so if you wanted to write one of these if you wanted to write the S interpolator we would maybe define a method that that would meet this this application here we wanted to find this method s so we do def s now this takes far args of type any you can substitute any type into into the S interpolator and you can have any number of them and any number of values can slot in there and this will take the seat ctx is the string context we get the parts that that's that's the name of the the value containing the fixed parts and we we use the interspersed method which has one serious disadvantage I'll explain in a second the interspersed method to slot the the variable parts in between that and then we just call make string okay and that will produce a string at the end of this so that the return type of this although I haven't included it will be just a string the disadvantage of intersperse is that it doesn't exist so you have to write something like this you have to first of all make sure that you have what what what we try to create here is a list of lists we want to make sure that the two parameters here are the same length now you've got sort of fence posts and fence panels here the we have to add an additional one to make sure they are the same length so we just prefix the empty string here and we're going to use the transpose method and then flatten it and what that actually does is it sort of puts the pears pears them up puts them in rows and then it reads those rows as columns and then ultimately what you get is all of these things in the right order there are probably dozens of different ways of doing this you could you could zip them you could write a while loop and append them to a string buffer these are all possibilities this I think is maybe the most concise but maybe not the most intuitive but pick your pick your poison hey so this this is all happening at runtime so this this is an ordinary call to s the the compiler has translated the string context code you write in your source to an application of this s so it's just like calling the method at runtime the creation of the string context object as well happens at runtime so a new object is allocated to the heap and so on now to make this work we need access to the string context and string context is a class in the scar in the scholar standard library we can't modify it we can't we can't magically get an S on that object but we can add an extension method here and we use an implicit class now who here is familiar with this implicit class I don't called a pattern but this this this way of adding extension methods to objects that are defined elsewhere yeah I think most people are sort of putting their hands up at different points and ultimately everybody is at least familiar to some extent good so this is how we make it work this is this is sort of the standard way of making our string contact making that application of the S on a string context work without the simplistic class in scope we can't do it now this is after the translation after translation in the parser this is a very ordinary method application it's dumb the translation does not know anything about the types involved all it knows is it's got a this thing that looks like a string literal it's got a little prefix which is some alphanumeric characters and it's got some some expressions which slot in this happens in the parser it has no more information than that so it doesn't do anything in intelligence the transformed code is subject to all the standard rules of Scala all the type system rules that would normally apply because it happens before it hits the type the type is where all interesting stuff happens in the compiler so we can define that method in any number of different ways provided it sort of meets the meets requirements for that call to s or dot whatever it happens to be to work so the parameters don't have to be strings then it have to be any they can be a type of your choice it can returns things other than strings it can return unit if you wanted to you could make it side affecting you can do all kinds of things there's nothing to stop you doing this it's it's entirely orthogonal you could overload the definition of s you could have a version of s that takes one parameter and returns a string you could have one that takes two parameters and returns a double and this this will work as if you had written the the transformed code directly into your source what people typically do is use varargs because often you want to substitute multiple things but that's not necessarily the the only way of doing it this method could have type parameters you could infer a type you can sanchit a type in the in the method application which which depends on the parameters that you substitute of types of the parameters you could add implicit parameter so remember it is just a call it can it can because the source code looks the way it does it doesn't matter if there's an additional implicit being pulled in the the actual application will look the same now consequently we have some unintended consequences so I under believe this here for a 30 seconds or so can you want to tell me what is happening here this will work in her in a fresh repple if you start up scala now type this in this is what you will get is anyone thinking what the hell is this so second so equals is a method which exists on the string context type good now we're substituting a value which is itself a string context into this now remember this this D sugars into creating a new string context with the constant bits which happen to be an empty string there and the empty string there so what this is essentially saying is string context empty string com empty string dot equals and then one parameter here which is the variable part which is string context this is a case class that it has an equals method to find and they are the same so we get true here I posted this on Twitter and people were a bit shocked and confused by it someone came back with with this this this also works and I little people didn't like it didn't like the fact that you can you can write code like this and it it looks weird I like to think of it as orthogonal like should we have another rule which says well actually these special methods these methods so I'm going to special case them and say these are forbidden in in a string context but everything else is fine or do we just have a simple rule the transformation is consistent every single time it's dumb you don't ever second-guess it some people didn't like my orthogonality thank my orthogonality claim I think it's debatable whether it makes the language simpler or more complex too oh wait maybe maybe I don't have any interview went to bills talk last which I don't if that would explain the difference between complexity and simplicity to make a judgment on this but it's not for me to say so let's see that's sort of the introductory level it's going to get a little bit more complicated now so we're moving to the type level and I want to talk a little bit about the motivations behind this why why do we want to do any experiments with these these string context now if any of you've been to my talks before at least one person here has been to this exact same talk before thanks for coming twice miles okay so yeah miles did miss the first the first half of the talk last time so I'm expecting him to leave probably halfway through or anyone else is also welcome to leave halfway through so all my talks are basically the same they are making it harder making your life harder as a programmer there at ways to make your life harder as a programmer to write code that doesn't work that I see is my role in life to make your life harder if you're trying to write stuff that will fail at runtime and we have we have dsl's everywhere and it's a it's a common part of what we do as a scholar programmers and there's this question should we should we externalize those those dsl's should we put them in separate files should we pass them at runtime or should we embed them in the in the Scala code and can we take advantage of what's probably the most complicated step the most I don't say intelligent the smartest step probably in our build process which is running the Scarlet compiler should we take advantage of that to use all the knowledge we have available during compilation to add additional constraints now this is a possibility with with the these sort of dsl's that we can put in string in interpolated strings so I'm very keen to take advantage of this information because more constraints is always a thing as far as I'm concerned so here are a few examples from mostly from from rapture so we can have a URI string context now in rapture this what this actually does is it reads the first few letters the bit before the colon and decides what kind of what type this should return this one returns an HTTP URL but if you put file in there it will return a file so we get a different type at compile time I'll them the methods I'll show you later will will maybe not directly explain how you implement things like this but you can you can sort of take the ideas and work out the various possibilities we could we have CSS we could we could find things like properties like color line height or whatever else you get in CSS and we could guarantee that the the name here is a valid CSS property name if you accidentally put a you in color which I do all the time it would it would be a compile error you would not accidentally deploy that to to to your to someone's web browser ultimately we can do XML chicken we can check that this foo here matches the foo here Semyon with jason rapture Jason does this load and we can we can check that this is this is well-formed Jason I've got a shell command interpreter so it will we tweak and we can sort of put some some variable parts with a file there which is the parameter to the WC command and then we execute that and we expect to get battered back an integer so we're sort of constructing here a command we're not running it but we can then later call dot exec on it and get a particular type out that's just a nice convenience method for farming work out to the the underlying OS email addresses that's pretty straightforward it can check with a ret regular expression there and we can even have things numbers with units we can check that's a number we can check that is a frequency we recognize and we could say that these two values are actually equal so if you have them we could have these these checked at compile time and it can know that combination of the but that the killer modifier here means that this is actually the same same value these these are the possibilities now i'm going to give you an example of why this I think is particularly well what the compelling reason I I sort of put a bit more thought into this recently I was working on the the scholar world website scholar wells a conference I ran in in the UK a couple of months ago and I was working on the website and I'm not a web designer i should say but I I approached the task of running a conference as a scholar program abidance of doing all of the things that are required in the sort of most scala way I possibly can and I i was i was writing some CSS which is as a bad start to begin with but I i was doing development work and at some point i deleted closing brace in my CSS so it wasn't it wasn't valid CSS and essentially what that meant was that the second half of the rules in my CSS i don't know how many of you have had the misfortune to have to write CSS but the second the second half of the rules that are in the CSS file suddenly were no longer applied and the result was the website looked like this now let me talk you through this um this is myles his dog this is john de goes who was doing training this is sort of the the venue there's some text here and indeed everywhere that that talks about the conference this this is this these the rules that were in the first half of the file they worked like I said I don't consider myself a natural designer but it's it's sort of strangely strangely beautiful in a in a kind of dark way right right so I thought could we actually do better this could we have scholar read CSS I don't want to design a DSL that is so has all the properties of CSS but requires me to sort of add capital letters in places and apply native scholar syntax to it i want to just copy paste my CSS into the into the code and add constraints and if I brought some HTML in the code as well I want to be able to link the two I want to know if I've got a class 2 i'll be talking to go on to a different slide let's can we get rid of it oh yeah I I wanted to take advantage of being able to cross-check a load of things in the in the compiler so this was the motivation I fixed that just in time for the conference but it did it did waste way somewhere time so the shrink context we've seen well why should I show you the beginning the s because and these these the ones were most familiar with these are the ones we see a lot of the time and they're very weakly types they did they'd to take a number of any parameters and they return a string whereas we would possibly have the criterion that we would rather embed to some some subset of types into there now 11 discriminating fact we would maybe use is if there's a corresponding type class this particular type i would like to be able to add that into the string interpolator any other type I want to be a type error this is this is a typical typical use case I think it applies in many of these cases we may actually want different types interpreted in different ways so depending on the type you may want the we may want it to substitute the value that has faced with the two string or we may want to for example put quotes around a string we have different requirements depending on the type so I'll talk about about embedding I'm use this word to mean embedding a particular type in a internet string and interpolated string so say we've got our ctx this is going to be our new string interpolator we can put an s in there foo that that makes sense we can substitute a string we can substitute an int I that that that will that we want Matt to work where is this one here where we substitute a future what does that mean what would we ever intend that to mean now i can write a type class that substitute strings i can write to type class that substitutes or embeds integers but I decided I'm not going to write one for futures now maybe they look like this so we've got this embeddable type class who is fine with type classes who is not fine with type classes okay cool if i'll ask that question two years ago i'm sure they're in more hands up the second time than the first time so i've got these definitions here they're not very interesting i'm calling to string on the end and indeed just returning the this is effectively that entity function here we can write this in scholar 212 like this which is a whole lot more concise this is using the Sam feature that's just been released a couple of days ago we can put a lander on the right hand side and provided the type the expected type is something with a single abstract method which indeed this one is it will also must be wrap that creating a new one of these so that's a really neat way of doing it I I would encourage you to do it but it's not what this talk is about now given that we want to have an embeddable type class associated with the types we substitute in here maybe a really naive way of doing this would be to define context as having some number of TT as an arbitrary type we're using fire args and just get an embeddable type class that corresponds to that T now could anyone tell me the problem with this so yeah so it works it works where all the t's happen to be the same but when you have a mixture for example like this list of 12 as a string and 3 is a double what's the best common super type of those it's any and what it would effectively do is resolve the type t2 to any and it would look for an embeddable of any we don't have one so we need to do something a bit better than that and this is this is my approach to doing I think you can do it with shapeless and H lists and a macro I can do without a macro so what I'm going to do is to find this embedding which will what will effectively do is for each parameter it will tie together the value that you actually place in the in the parameter position with the corresponding type class so every single value that gets put into that that method call will have the type class pulled in different one potentially for each different value this is so it's creating a new embedding that is where it pulls in the embeddable type class and that is the value that's where T comes from that's where it knows which embeddable to get and we end up with an instance of embedding and indeed here we we ask for values varargs of type embedding and the type here is existential so it's they can be from types and what we do is we map the apply method so we can write that very concisely like that apply is defined as basically getting the getting the type class this one here and calling embed on it so this is a neat way to tie tie the two together and avoid difficulties with with loves so this will now work it'll get different type classes for each of these this will not because I never defined one for double and it will will get a type error at this position here and it'll say can't find well it'll say type mismatch but you'll know at least where the error is is that it's that kind of okay is anyone uncomfortable with that don't be shy okay so I thought what about that version works for the embed effort that's just one particular type class so I thought I'd write a more general version so we invented this thing called a next it's a generalized version of the the embedding here so I'm going to I'm going to take a type class which is a higher kind it it has it this is this is a type class type so it itself takes a parameter but we're not going to specify what that parameter is so this might be something like show if you're familiar with show from cats we've got the type of the value and then we're going to be able to access the value and access the type class from an instance of a next oh my battery is running low this could be a disaster asura is rip our outlet nearby thanks just went plugging it in thanks okay hopefully we are all good and we have an implicit which actually creates a new one of these so if it sees a value of type T it creates a new and it needs an A next of some type class it will it will apply the Scala compiler will realize as a type mismatch and it will apply this implicit which you can see most of their I think that's that's pretty much all of it and it will it will create a new a next and what's probably the best thing for me to do is show you how you use it so forget embedding we're going to use a next now and I'm going to ask for varargs of a next of the embeddable type class remember this this this boilerplate around here is just the bits we need to make this into a string context and then given given the values here for each one we're going to call dot embed on it now this is like a lambda within a lambda the implementation if I go if I go back I didn't show you the slide before is we've got this a double lambda here what it's saying is effectively for each for each value we call apply on each one which takes which itself takes a lambda and that lambda is the method we want to call on the type class that has been resolved for that particular a next value now all of these because embed in our earlier example always returns a string even though the types of all the all the annex a next's annexed vary their their their existential we end up with a consistent type at the end so we end up with strings here you can write it like that I would write it like that but anyone i worked with would probably dislike me a little bit more for it so what this means is that we can now write things like interpolating strings like a trouble that that works fine this will fail unless we have an implicit too fine for dealing with embedding classes and this is exactly what we want so this little trick you can actually that there's no there's no reason why this wouldn't apply to other other methods that are completely unrelated from string interpolators it's just particularly useful in this in this example we have a bonus level that's that's the end of the type level by the way but there is a little bit more tug level stuff later but now we're going to go for a bonus level I was some I was up late one night reading the scarlet source code in bed as I I have trouble sleeping very very good very good I like I was reading passes that's actually looking at the the source code for how these string contexts are created I noticed this little comment here documenting that it is intentional that the ident is not rooted for purposes of virtualization and there's another line of code which was commented out which was effectively the the code to modify the ast to insert a route scarlet string context but that was commented out and instead it was inserting a string context so not route scarlet or string context which is completely unambiguous it was inserting a new string context so the transformation is indeed exactly as I described it on the third or fourth slide it's not creating the exact class you think it might it's creating a new instance of whatever happens to be called string context and is in scope at that time and this is and this is this is intentional so of course this got me thinking what could we use this information for because I mean we've got we've got the implicit class method we can we can add extension methods and we get that functionality but one thing implicit classes can't do you can't you can't use an implicit conversion to convert something from some type to a dynamic it will not so I by the way who who has not seen or heard of dynamic in Scala you're something you're looking at each other to satanic quotes that be you're not putting your hands up as everyone that's everyone's seen you haven't seen dynamic oh so dynamic allows allows you to call a method on an object in in in scarlet if that object inherits from dynamic trait you can call a method with any name you like any identifier and it will compile and it will compile to a call on that method to a method sorry on that object to a method called select dynamic which passes the identifier the method of the supposed method named as a string literal so it's sort of lifting from a what waterboard appears to be a method that you expect to exist to a string literal with that name so I use this in in rapture Jason source of how about some Jason valued foo there's no method called food to find but it works as if there's a I mean it's like a JSON object that will try to access the foo key in that that that that JSON object and it works fine i mean i quite like it i probably use it more than more than most people do but you can't you can't do an implicit conversion so that i mean if you could do the implicit conversion then you could actually have all method calls you could convert from all T to a dynamic and then you could just call anything on anything and your scholar would suddenly become an entirely dynamic language and one would fail but how about we create a dynamic string context and we call it string context that's the important thing if this is in scope we can define apply dynamic this is the name of the method call so for an interpolated string this is the prefix so I might as well call it prefix there and then this is the the arguments that get called on that on that method and I'm going to really really simple in from implementation here and just return the prefix whatever that happens to be I'm going to completely ignore the variable parts and I'm going to ignore the constant parts and it means you can write this you can write anything prefix any any prefix then an empty string and then it will just it'll it'll create that apparently only got five minutes this is something is to someone following me oh ok i thought i was the only person in this slot so let's rush through level 3 the macro level no miles might have thought of something since we discussed it yeah so you can't you can't stick a space between the prefix and the the opening quote i mean what you could do is have like a different kind of symbol in scarlet that instead of starting with a single quote always ends in a double double quote but you're welcome to come up with better better better ideas for this so i will i will rush through this macro level which is probably the worst thing to do because it's it's the level above the type level imagine we want we want to embed values in in in Jason we've got an F value there that that would make sense if it's a string or an int that's fine but I mean we could write that or we could write that and when we substitute that that value in if it's there if it's a string then we want to put quotes in as well if we replace that with the string in quotes then we end up with too many quotes and it ceases to parse so what I how I would describe this is the context of this this whole is it's different in these these two examples and you could even imagine something like this where you could if F were a pair from straight string comma string then maybe this would make sense but I mean we're treating it differently in each case and this this is obviously something more complicated than simple string substitution so let's do this with macros because why not and remember like I said at the beginning this string context are all orthogonal to other features of scarlet so there's nothing to stop you implementing this with implementing the the method with a macro so if you remember this would get translated to something that looks like this we're going to pretend that we still got the the all the annex stuff I would implement this jason with I've got this to Jason type class or I'm going to get varargs of a next of that so any type that has a to Jason type class can be substituted substituted in there and we will implement that with a macro and the approach which I will sort of go through very quickly because I don't want to make this a talk about macros least of all in the last three and a half minutes what we were what we will essentially do is take advantage of the fact that we know that the transformation that happens in the parser will be consistent all the time it will always it always be the same shape and we can match on that AST it will look something like that these will be the literal values the constants the constant strings we compile time we don't know what the-- we know what the expressions are but we don't know that don't know what the values of those expressions are but we can lift out the literal values there they will look like this little of constants they will be typed as strings so essentially what we do is at compile time you have access to the values as just a list of strings of the constants in our string interpolator an interpolated string at compile time and then the macro can can use that it can parse it it can it can check it it can throw compile exceptions it can give you warnings these are all these are all useful things you can do with it I'm going to skip all the type level stuff unfortunately but you can you can do some interesting things with H lists and singles and types here is a here's our code sample now remember this gets d sugar to the string context and this is the this is the what the implicit does it calls annex on f and remember typer is looking into this it's a works out that it needs to apply the implicit value there by virtue of this implicit being found it knows that the type parameter to annex are these and this results in a typed value here and this is all sort of an example of what the compiler is doing particularly the typer and it knows all of this information it knows that the value here is a string now I don't really explain how but we can actually get additional information in there as well we can we can define our type classes so that we know that the context for Strings being sorry yet the context for Strings being embedded into Jason can be some type some phantom type with with seconds to go this is the wrong time to be explaining phantom types now we can use that information to make something like this work we've got a string we've got a pair of string and an int we've got another stream there this can be checked at compile time it can know that the the context of this substitution here the context of this substitution here means that the way they get put into the into the code is is different we can know that pair by virtue of the type class that is resolved can be substituted in position here but could not be substituted here I mean this is code that should compile so we can do all these checks in the macro it can it can do some very interesting things and we get we get more safety we are making it harder for you to write bad code one last level the boss level which I try to explain at scala io in in in leo a couple of weeks ago and realized that i lost miles he was it was my last my last remaining person who I understood everything up at that point I lost him there and I realized actually I didn't understand it either I I i discovered this the day before I III went to winter Scala I Owen Lee on the day before I visited epfl in in lausanne which is a couple of hours away and I made this was a terrible mistake because i went to see Dennis chef Alan and the worst thing you can do when you've got a almost finished talk is going to meet someone who knows more about the subject than you do and I had a chat window his office had a chat with him about it and he said are well you can do a load of other things you can do pattern matching and he showed me how you do pattern matching now you can imagine we put a string here and we can we can write a case clause here and we can put a interpreted string in here which has I mean it's got some content it's got I mean I've put this variable name here adjacent to these parentheses what I want to do is say this this is a this is a regular expression for reading a an HTML tag and I want to label this this capturing group here and I want to call that tag and I want to call this one this one here text so you can pat match on a string you can get the contents of this one here as tagged you can get the content to this one here is text and you can create a new instance from that the code you need to write to do this is so horrendously complex and that that's that's a very sketchy sketch implementation but the best thing to do is to go to dennis's github repository if you want to take a photo this it's it's very well documented there's a very very clear explanation of how you can write this but it is and it's step by step and i would recommend if you're interested in doing this sort of thing in in in pattern matching on interpolated strings then it's well worth following that and that that is that is all I'm going to do on the boss level this time maybe maybe in the future I will understand it too but for now there is that that repository and I'm out of time i'll say i'll say thank you and I there's probably no time for questions right yeah so thanks everyone [Applause]