SBTB 2014, Jon Pretty: Fifty Rapture One-Liners in Forty Minutes!
Recording: SBTB 2014, Jon Pretty: Fifty Rapture One-Liners in Forty Minutes!
all right thank you thank you jason hello hello scala by the bay you having a good conference um come on that's more like it right um so i don't know if any of you saw the the presentation i gave at scholar days earlier this year you have a show of hands anyone who saw it so not many people okay um if you had seen it you're gonna be like grossly disappointed by this one um it's not not nearly half as good so setting that expectation i'm gonna be talking today about rapture which is uh a collection of open source libraries which i've been developing for the last few years now when i when i give these presentations i'm always looking for advice and and help on on ways i can improve them make them better so i went to my um my good friend paul phillips uh and he said no he did paul certainly did not say that paul did not say that uh that that's not uh in his vocabulary he said yeah really if i thought this situation would make much more sense if only john pretty were talking over it thanks thanks paul so without further ado sure okay enough of this yeah you you didn't you didn't get that there was maybe miles who was just like peeping in over the corner there the uh the screen dimensions changed anyway i'm going to do this really fast we're going to go through hopefully uh 51 liners uh in in rapture in uh i've got about half an hour left i guess um so raptor is um first of all an io library um when you're working with io you want to talk about resources and like places where you you have data so one one such place might be a url so this is an example of how you write a url in in rapture io so there is the domain name there is the the path and if you were to type this into the reply that's what you'd get you could do the same thing with a with a file so if it's on your local file system that's kind of what it would look like but this is uh this is the old way of doing it in in the latest version of rapture we've got a a uri macro which is a string context so if you've seen string context before like the uh like string interpolation you normally have an s there this is this is kind of the same the same same feature of scala 210 but this actually invokes a macro now we'll notice this is i mean it looks like a string but it somehow worked out that it is an http url rather than some other kind of url um so this is this is typed as as an http url which okay that's not so impressive until you see an alternative where the only real difference is that it starts with the word file there so raptorio gives you a really neat way of defining resources uh uses white box macros to to work out the exact type so you get all the appropriate functionality for that thing without without having to learn the old slightly slightly convoluted syntax you saw at the beginning so one thing you might want to do is load an http url from just a plain old string because we don't we don't know at compile time what that string will be so if that's what you wanted to do you would simply go with http.parse and then then string and that will hopefully return a valid http url i'll show you later on what happens if it turns out not to be valid but for now let's assume we'll get an http url out of it i'm going to turn the uh the volume down there we are so one one feature of raptorio is that it it separates capabilities now capability might be for example as we've got here slurping the entirety of resource into memory it separates that capability from the definition of the resource so this is the resource nowhere on the resource is it defined what it means to slurp that's not that's not part of the interface the interface is pretty empty in that case slurp is is uh is added on uh using using an implicit conversion and if we if we call dot slurp on this thing then we get some bytes so those are some bytes by default it will display them as um as hex why isn't this an array of bytes did anyone think of any disadvantages with just returning an array of bytes whoever said it's mutable is correct i didn't hear you um so this is this is like an immutable wrapper around around bytes in the same way that we have a string for character data now this is this is uh this is a json file we can we can assume so we probably don't want those bytes we want a string so in order to do that all we have to do is change the type here or not maybe okay we've got an error so we've tried to slurp characters from this this exact same uri now the error message here what does it say error cannot find implicit reader for uh file url resources file url resources can only be read if a reader implicit exists within scope that's i like i don't really know what that means uh note if you're working with uh character data you will require an implicit character encoding eg import encodings.system or encodings.utf8 okay well that's more useful now this is the typical approach i've tried to take with rapture it makes use of implicits very heavily now implicits are great but they're meant to kind of be hidden they're meant to they're meant to be uh um i was going to say seen and not heard but they're neither seen nor heard so as much as possible i provide additional information about what you need to do if there is an implicit missing within scope so here it says import and encoding so let's do that this is all we all we have to do now one thing to note here is i've written this as utf-8 in backticks now a lot of a lot of libraries certainly in java uh will will apply some kind of arbitrary transformation to identifiers that that can't well i identified include a character that can't be represented in a java identifier such as the the hyphen there uh now it's a pain as a programmer to have to remember what particular encoding has been used to do that so in rapture i've said okay well we'll just use whatever the canonical representation is and if there are if there are characters we can't represent we'll stick it in backticks and it's a simple rule and it applies everywhere that there is that issue so having done that we can then we can then slurp this in again as uh as character data and we've got some we've got some json here now this this particular file i think it is actually still on the server there if you want to have a look but it starts off with it it's basically um a very brief description of the the 2012 election um this is a string so it works out that given given characters as our as our as our type for for data to be slurped a string is the result in the same way that bytes result well a a byte type there resulted in the type bytes coming out here now rapture is also a jason library so given that string we can parse it just do json.parse and and and the string and we get something of type json now with for example the standard library json parser you end up with with an any type i mean it could be a could be a map could be a list because json is dynamic we don't generally know at compile time what what what types it contains and that that's that's like a fundamental property of json uh we we we just uh we we just don't know unless we've applied some schema and and right here we haven't we haven't given it any reason to believe it's it's a map or an array or or a scalar value but it's it's able to print out um print out what what the what the json is it's parsed correctly we haven't had an exception thrown so that's good now in in the previous one this is this is uh this is the uh the slightly older version of raptor jason you you have to get get the string first and then pause it in the latest version which isn't quite released yet should be should be after the next couple of weeks we can just put the uri there and as long as that uri is readable so there is the capability available as a type class for reading that that kind of uri now remember this this is an http url it's not a file url there might be different rules for what what is readable and what isn't but this http uri is readable we can just put it in as the parameter to the jsonparse method and it will pass it directly so that's that's two lines in in in in rapture zero nine zero and uh and down to one in zero ten so what can we do with this jason well if you let me show you the previous one we have we have a map and there is this uh this candidates element which is an array now i can just call dot candidates on that now this this might seem a bit magic because i've already said that we don't know anything about the schema for the json but nevertheless i'm calling dot candidates now this is using scala's dynamic trait so at this point i could call dot anything and what it would do is it would assume that there is that type there that there is the the value candidates and it will effectively dereference it and when it when it calls to string for example in the repo here calls to string we get the uh we get a string representation of the candidates likewise if we wanted the first one we just apply zero to it and it plays a string for the the zeroth or first element of that of that array in the json within that we can call dot name now you might wonder well am i just am i just exchanging loads of arbitrary casts as i would have written in the old-fashioned uh in the old-fashioned standard library jason parser for slightly nicer syntax but still all the same risks of of typing well actually no this won't ever throw an exception i mean two string may fail to return a useful value and it'll it'll say undefined if it if you uh if you try and access something that's not there but you won't get an exception drone the point at which you get an exception is once and once only at the end of accessing a particular piece of data within json so when you call dot as and dot as is pretty much the only method defined on a json type when you actually try and get a type string in this case from that that that path into the json that's when that's the one point at which an exception might be thrown um so in in this case we've managed to get the string out of that json likewise we could have done dot age and dot as int and it will it will grab an int so that's that then that kind of works as you would expect now something more complicated is i mean maybe we want to get like an entire candidate out in one go from from our jason maybe we want to represent that candidate using a case class so the two parameters we're interested in are the name and the age so first of all let's let's define a case class to represent that that should be quite straightforward by the way this is meant to be completely interactive so if anyone has any questions just yell out and and ask oh by the way can you also count how many slides i'm going through because i've got no idea how uh how close to the end i am we're on about 20 at the moment i think maybe a bit fewer uh so we we can we can actually say dot as and a case class type here rapture will extract well you can extract using the az method any any type which it knows how to extract now that includes the scalars like int double string and so on it includes any case class so it uses a macro to to associate the the map values name and age in this particular case with the with the parameters to the case class and if they're present then it will be successfully able to extract a candidate so that that gives you a very very quick way of quick way of constructing some data and effectively applying a schema to your json now what what is this return try now i i mentioned that if you if you try and access a value that's not there in the json then an exception might be thrown who likes exceptions being thrown so there is one hand up thank you for being honest exceptions aren't always the best way of dealing with failure as you may have experienced there are alternatives so you maybe want to return a try now what i could have done is i could have re-implemented this whole api with different variants one for returning try one for just like throwing interception if there's a failure maybe some people wanted to return options or either so i've mentioned four different versions of the api this this is becoming quite an overhead now but how about how about just a general solution so what you can do is you can import a mode you can import return try that that is an example of a mode and then as if by magic that exact same call now this is literally the same exactly the same code you saw on the two slides ago it is now returning a try of a candidate it won't throw an exception the old the older version would but because we've imported that modes dot return try this will magically and when i say magic i don't mean macros this is this is uh completely independent of macros um but it is magic so this is hopefully this hopefully gives you a lot of flexibility um i i could talk for ages about about modes um and in fact i did that at the scala workshop in uh in sweden a couple of weeks ago um there is a presentation online i will i will tweet the details later on on that but it explains how modes work and gives you a few examples of different ones for now i will i will show you that one and another one which which kind of arose by not not so much by design but by serendipity um i discovered that actually you could you could return a future instead so rather than rather returning a try or an option you could just import this one thing well you import this in your also important execution context but given those two things in scope any any fallible method any method which might throw an exception in in rapture will actually now return a future and that will be executed on the on the thread pool all it takes is this uh this import so for example with that in scope we can just take our uri we can slurp it this is this is a risky operation i mean your your network might go down anything could happen it also might take a long time you might have a really slow connection so let's let's instantly return a a future of that result it knows it knows all the details of the the return type and at some point when that completes if it completes um you you'll you'll get a a string inside a future so you'll get ambiguous implicits um yeah so it needs to find a unique instance of a mode in scope by the way anyone tweeting me will find that actually the whole podium vibrates maybe i'll put that down there now the whole stage will vibrate so um yeah that's a good question you can have only one one mode in scope at any one time um but but they are they are scoped to well the implicit scope so you could have different parts of your code that use um use different modes and it might be really appropriate to have one mode for accessing json and another mode for for doing your i operations question over there how confused would i be if i accidentally organize imports um i i i so does does your organized imports command is it uh is it aware of implicits well yeah it's a word i think it's aware of implicitly for intellij but is it i would i'm just it's a new thing for me to think about it right yeah so the the the ordering shouldn't the ordering of the the import shouldn't be significant if you organize imports in intellij it will pull ones out to the top generally speaking so you can do an import statement further down in the code if you just wanted a particular function you want to have it behave in a certain way if you do organize endpoints it will rip it out of that panel unless you've done something funky in your settings and if it's if it's not explicitly referenced right if you're typing yeah i mean so you're right this this isn't explicitly referenced um thanks thanks i'm glad someone here knows more about intellij than i do um yeah question front easier is because so where you explicitly specify what you want that is actually an option yeah you can um if you import instead of return try or return futures you can import something called explicit mode now that will mean that everything will uh return what's called an unevaluated value and that has methods on it like um uh it's got like attempt which will return a try it's got future which will return a future it's got option which will return an option um so that that is a possibility yeah where you want to where you wanted on the case-by-case basis uh i'm going to say last question on modest you want to have a future of a try or yeah you can it's a good question you can also compose modes so they are an implicit and normally the easiest thing to do is import it and then it's just in scope but what you can do is define your own your own implicit mode and you could say implicit mode equals implicit val mode equals um and then return try compose return future and that will that will return a try within a future yeah i should have talked about modes today shouldn't i right so another thing you can you can extract is a list of things so there was uh there were in fact more than one candidate in the election and if if we if we do json.candidates.as list of candidate these things are composing as well so we can in in one line we can extract both candidates where it it is it is effectively um uh getting getting the the string and and the inch from here likewise stringing in from there which enables it to get the candidate in in both elements of the array and then construct a list of them so lists are nice but other other collection types uh also exist who likes using stacks um probably one of the the lesser well-known uh collection types in the standard library but you can extract them so you can you can in fact extract any type uh any collection type that is buildable so it has a can build from available so yeah whatever whatever collection type you want to use just just specify that and that will that will pull out the uh the right type and build build it for you i've defined something else here i've defined an entire election so my election has two parameters there is a year i didn't you didn't actually see the year in the in the json example there but but there is a year value and then there is the the sequence we already saw of candidates so all we all we need to do is specify our election as having two parameters candidates and year and we can we can try and extract this and it will actually work so in in one line plus plus some case class definitions we can just take the whole the whole of our json structure and pull out an election from it and that is that is what it looks like so we we've we've effectively used a schema defined by case classes and pulled the whole lot out and this this works for any case class provided all the parameters of the case class are themselves extractable so applying this recursively elections are extractable because um because essentially the candidates are and the the string and the inter are extractable we can also construct some new json this is if you like a json literal it's a really simple maps got one one key and one value oh by the way this this says nothing about my political affiliations it is merely an example don't don't infer anything we can even we can even substitute values in there so in my in my new vote json i can i can substitute uh in the same way that you do with an interpolated string just just put a json value in there and it will will substitute it in so any any any type which is extractable can be um can be put into a json string context like this so this this allows you to very very concisely return a chunk of json so we we have we have a an immutable data structure representing json one thing we can do is is modify it or well at least we can we can create a new version based on some change so this here is a effectively a lens we can we can say well we given that given our existing vote object we're going to create a new one with the year value set to 2012. so that wasn't there before we just added it and this is a this is the definition of uh of that updated json now this might look weird does anyone know what's going on here so this looks a lot like the um the string context i showed you before but it's inside a case so what it's actually doing is is pattern matching on jason on a json literal so if your vote and you saw the vote before happens to be a map with a vote key inside it which itself has a name key we can extract a value c which we can then use over the right hand side of the case clause so you might want to use this if you had a really complicated json structure and there were like two or three values deep within that structure that you want to pull out and use in a in a case match you define only the bits you're interested in and and you can literally just pull them out and and they are available don't forget when you pull a when you pull a value out you don't know what type it is so you still have to call dot as and specify type c is of type jason we we still have this this challenge of getting from the untyped dynamic world of jason into the scala world and and the method as is what what takes us across that boundary now you might have assumed probably quite reasonably that i i just implemented my entire jason parser and uh and then an ast representation um i didn't feel confident to do that so i thought okay well i'll just use somebody else's and the best by far jason library out there is the stat hold on no not the standard library um well there is there is a json parser in the standard library who who has used it who has stopped using it so that i think i think it was the same set of hands up both times it is there it works but it's not particularly fast and uh i've heard tales of memory leaks and all sorts of nasty stuff but you you can you can use you can use as a back end the standard library parser and the benefit of that is at least in scala 210 you have no external dependencies if that is a primary concern above like other stuff alternatively you might want to use jackson i'm going to i'm going to mention a few of these so um can everyone cheer if i mention a a library a json library that they have used so uh jackson yay okay a lot of people like jackson jason for s so a little bit less but but jason frese is supported lift jason yeah good good argonaut yay well there's definitely a sort of scarlet z crowd over here um does anyone know john you should drawn is uh erik osheim's jason library and it's really fast um it is possibly faster than jackson at some things you might not believe me but it is so all of these are supported they use type classes to basically abstract over the over the parser and the ast and it just means that you can you can slot in whichever whichever particular back end you want to use so that that's immutable jason and in the same way that we have lists and list buffers you have jason and jason buffers so what we can do is we can start out with a pretty uninteresting json buffer an empty map and we can just add a value to it so we just say jb.life we're gonna we're gonna um oh there's a mistake there that that should say jb.life equals ignore the plus there and that was that will set the value of life to 42. within our with our json buffer we could same mistake again sorry we can add a candidate uh to to our json buffer there it is this this is all we have to do so this this is actually kind of like a uh probably a very very slow and inefficient but in memory database we're just like modifying mutating stuff in memory um given that given we we've made all kinds of mutations to our json buffer all of which are typically one-liners we can say write that at least in version 010 we can say write that to uh to a file so we're going to write that to home data candidate.json uh jason at the back five minutes okay i'll i will how many slides have i done so 010 is the version of rapture jason yeah um so i'm probably about 42 through so i'll i'll do the last eight or so uh this this isn't the feature of the currently released version that's why i've specified zero 10 there previous slide are you trying to influence us no no no not at all she hasn't even declared her candidacy yet um yeah yeah she happened to be well there are no republican candidates yet honest so uh oh by the way this this one this works because file urls are writable if we were to put an http url here it just wouldn't work because there's no way of there's no way of writing to an http url i mean what you could do is you could you could define a type class and implicit within scope that says well i i've got a way of writing to http urls i i've no idea what it is but if someone were to provide that type class then you could you could specify that and and here you could have an http url a question over there sorry put is an http command which is also available in save um yeah you could do yeah um so i'm i'm very open to suggestions on on on the api thank you we've also got copy too so we can copy from a file to a file so there is our source file candidate.json and well we're we don't want to risk losing it so we'll make a copy on our file system so there is the new copy and the generally methods which return unit as this might do are sorry save as save as so this this one is more like a copy but i mean you in the previous one save as this one okay yeah you could do um what whatever so we we can copy this to from one place to another and yeah as i said uh unit being returned as a as a result type is pretty uninformative so it would be useful actually to have a little bit more information so you get this thing called a summary a copy summary which says using a file system copy 85 bytes have been transferred great okay now we can we can copy from http url as well now all you need to be able to copy is something that's readable and a destination that's writable anything that's readable can be copied to to anywhere that's writable so this works perfectly fine as well but the only reason this works is because raptor knows how to stream so this is a streamed copy this is not a file system copy so the file system copy will be quick it just runs the command on the file system and says like copy copy this and it doesn't have to load the data into memory and write it out again we don't have that luxury with http copying to files so it kind of falls back on the next best thing and does a streamed copy so you might be interested to look at the this actually statically resolves so it doesn't know at a compile time whether it's going to stream or or or do a genuine copy um so this this works fine too we can even do something like this so we can copy straight to my class path this is this is a a class path uri that works in in rapture 010 and we can copy it straight to s3 i haven't actually implemented the s3 thing yet but it's it's pretty straightforward as long as we've got a way to write to s3 and you could provide that as a type class in one line we can go straight from from class path to to a writable resource somewhere and again you get the streamed copy summary oh i've got a whole section on cryptography i'll go through this really quickly so you can you can generate a key you specify type here as to what um what what what kind of crypto you want to do uh we can do blowfish so we've generated a key here with a key we can encrypt something so let's encrypt a string we then get some byte data out again this is actually a subclass of of bytes encrypted second i thought someone said something was the heckle okay um so yeah we've got some encrypted data here maybe maybe displaying those bytes as hex is actually a bit bit verbose we want to use base64 instead so we can call as this is this is a different as method from the one you saw before but it's kind of got the same effect we can display that as a a string of base64 characters and then having done that we can decrypt it again notice that because that that encrypted is is previously typed uh back here as as encrypted data of blowfish it knows that well and also the key is is typed with uh with blowfish it will it will check that the type of the key is the same as the type of the encrypted data and it knows at compile time that it's safe to do that so we get this decrypted uh byte data back well that doesn't say hello world because we don't actually know that it started off as a string it got serialized somewhere along the way to bytes so once again we have to say as sorry it's a little bit covered up there we say as string and then because that happens to be an encoding remember i imported encodings.utf8 because that is in scope it is happy to transform those those bytes into a uh in interest string um so a decision i took uh quite early on it's relating to what jared was talking about earlier that you end up very frequently with encoding issues uh if you're not careful java uses as a default the system encoding so your your code may run absolutely fine on your local machine you deploy it to a server which has a different which maybe has iso 88591 instead of utf-8 which you're using locally and stuff breaks in subtle ways so at all times i i'm i insist that there is an implicit sorry there is an explicit implicit specified whenever you do anything that requires translation between bytes and characters such as here now this is kind of cool so you know i showed you all those different back ends you can use for for json i thought wouldn't it be interesting if rather than actually use a json parser i just plugged in an xml parser instead because i can use arbitrary asts so okay rapture xml was invented and you can you can parse some some xml like that and it knows how to pass it you get an xml object which is pretty much equivalent to the json the structure i had to modify a bit because the structure of xml is more complicated than json but we can do all the same things we did before we can call dot candidate dynamically we can get the first element we can call age on that and we can extract an inch from it and it all it all works reasonably well there's a few there's a few corner cases which which are more complicated than they are with json so there's small work to be done on that we could also do beeson i haven't i haven't done it yet but beeson is probably an easier task than xml because it more closely represents jason okay so this is this is a one-liner i mean it spreads onto several lines but uh it is it is um it could be it could be compressed onto a single line so what what is this doing i mean you probably wouldn't want to do this in a single in a single line but it's possible so okay we're using a key the one we the blowfish key we had before and we're encrypting some json which we're parsing from straight from this url now this will give you give us an immutable json object and we are adding a password to that which is secret so from from here sorry from from here up to here is a it's a new json object which i've i've created with with some modification i've encrypted the whole lot and then writing it or save as if you prefer to to this this url here which is a file this is writable so the whole thing results in a write summary which is the the size of the result well tells us the size of the resultant uh resultant right so this is my grand finale this is like trying to combine all those different things into into a single one-liner hopefully that's that that's interesting and i'll leave you with um paul phillips follow-up quote in spicewood including john pretty talking over things this could be an improvement i i hope it i hope it i hope it was an improvement uh that's it thanks everyone you