Your Type System Working For You!

Scale By the Bay 2018: Richard Wall, Your Type System Working For You!

Recording: Scale By the Bay 2018: Richard Wall, Your Type System Working For You!

you good enough okay thank you very much sorry about that okay so the earth it's not flat it's also not a sphere okay it's a lot more complicated than that I'm gonna race through this because I'd normally have a lot more time for this talk you'll just have to believe me on the earth spins it's fatter in the middle beyond there that's like level level three understanding it's not flat it's not round okay it's also not even a lip sight because there's like different densities in the earth and things pull on the crust in different ways and stuff so the upshot of all this is that it's really hard to give a location on earth in like a standard way the way most of us are used to it is GPS Latin locks right this is what you're getting a car any GPS system and it's a pretty good system choose a center point that center point will actually move around depending on the model you're using because again not exactly spherical and stuff but take that draw one angle out from the meridian that's the thing that runs through in the middle of London draw another one up from the equator and draw a line out to where it intersects with the model of the earth and that's a point that you've gone on the ground okay that works pretty well except as you go further north one of the longitude degrees changes how much distance you move that makes the math hard there you can kind of see it here that the squares get closer together right so there's also two types of primary data spatial data that we work with I forgot to say this but way back in the mists of time I got a degree a master's degree in geographical information systems and I used to work in that for five years I took a career change and then just recently last three years or so it picked up I started doing it and be that the the discipline hasn't changed much but the tools and the languages have and it's wonderful what you can do now so two primary types of spatial data this raster and vector vector is what you get on your GPS again lat/longs draw lines in between Philemon with colors you get a nice scalable map it's like SVG raster is like an image okay it has this assumption that all the pixels are the same size and that's the problem if you go back here you could pretty much see these pixels aren't really the same size right as you move around so why does all this matter it matters because we have loads of different ways of taking something that's round like the earth and squashing it flat to put or to make data processor ball in 2d and that's really what we're aiming for there's all these different things conical projections there's polar projections this one this is a conical one the other one besides lat/longs or wgs84 that we deal with a lot is this thing called a universal transverse Mercator it's a cylinder that wraps around the earth and there's like 60 of these that go around the earth they change every six degrees and you kind of squish it out and then open it up again and it makes something approximately metre based and flat this is the US and in the mainland lower 48 there are nine there are nine different UTM zones that covering okay right this is not our talk just on UTM s or CRS is but what I do want you to take away is what happens when you get the CRS wrong and this is the key to this talk okay this is a field in UTM zone 15 it's defined and you can see that it's a field right the outline of a field if I get the wrong CRS but then if you can see it but deal here that's that field and it's now in the middle of Lake Erie okay and that's the worst possible come we didn't get an error we didn't get no data we got a bunch of water where there should be a field and now if you try and process that you may not even detect it if you're doing massive scale right you're not gonna go and check every field that you process and you've just given out the wrong results some farmer somewhere is scratching their head going why are they saying this thing water in this field so so what is most code do most code up until now the state of the art is lots and lots of redundant coordinate reference system checks you make sure that the coordinate reference systems are the same or as expected when you when you use them and it looks a lot like this you'd have some kind of thing that says ensure the same CRS and here's our two CRS is and we do some kind of comparison against them and we throw an exception if they're wrong and we rely on those runtime checks to make sure everything's working now this is where I think it gets interesting because this is where scholars tight system which wasn't really available to me when I was doing this stuff before really kicks in Scala has this concept of variance between types and subtypes and a great under estimated variance is invariants I really like invariance because when in Burien any type parameter changes the overall type and makes it a completely different type in the system so an example of this is set ok set an event and setup strain are by scholars compiler standards completely different types they have nothing come on okay you cannot that you can't pass a set of int into something that expects the set of any it's not co-vary it's invariant it's a different type so this is not a talk on variance but what you really need to know is that list is covariant if you have something that says I've take a list of any you can pass it any kind of list and it works with a set set is invariant if you pass it if you say it's a set of any then what you pass it has to be a set of any if you pass it anything else it won't compile okay so given that let's see what we can do with these CRS is to disambiguate them start with a simple definition I like case classes I think they make an excellent basis for an API I always start with the case class if I can so here's an abstract cast first of all and this is going to be our coordinate reference system definition it takes a key RS code it does some kind of look up this this mechanism if you're interested we use a library called Geo tools it's a open source library has all the definitions for the standards the epsg standards which is thousands of different CRS and you can bring the mathematical model in for them so when you do this there's actually a mathematical model which includes the datum the projection all of the information that's needed to define that coordinate reference system we can create a case class that extends this definition up here now you might be wondering why case class and not case object here right so that's the next bit we need that case class and its companion so that we can do a bit of tricky stuff I think this this may be the first bit of real coding where you might say oh I didn't know you could do that so here's the case class it extends this epsg for 3 to 6 that's our lat/longs that's our wgs84 the object extends a companion well it's a it's a abstract class that is int in intended to be extended by companion objects okay it's also a tight class hopefully anyone not really not know about some classes do I need to do I don't think there's gonna be time type classes are cool you should go look them up if you haven't already okay so this thing CRS type is a tight class it's a class with information about this type here and specifically it's the definition of that by doing this we can create this abstract class CRS type that's what this extends we can suck the type parameter into a tight memo I love this trick okay we've taken something that the Schuyler compiler has said this is the type that we've got and we've shoved it into this instance and now it lives in that instance and the compiler can pull it down again later there's an apply method on any companion object and we know this this apply method is going to produce a CRS def so this is a abstract but we know it's going to be there by some by some companion object that extends this will get that since we've got that we can call it and get the instance of our CRS definition from that we can put out the actual coordinate reference system the definitions we could pull out an ID if we want to and then I love this little tree right here you might Michael might not have seen this before so a companion object is a low priority it's a place where you can check for low current priority implicit definitions given some type here a PSG you for three to six if there's some implicit that needs to apply to that the companion object will always be searched for that will be searched last but it will look in there by making this thing have a pointer back to itself and marketing it as implicit this companion object becomes its own implicit type class okay because it's inside itself first time I did that I'm like I did not know you get away with that I thought I was running away going I got away with something this is awesome so this means what we've done now we've created a definition we've created a companion object for it and these two things this is now a type in the system this is its companion object and then linked together so what it means is we can do some things like this I'm going to show you some of the some of the stuff that this gives you let's say we've got some kind of geometry type a point or a polygon or something like that this is now tight parameterised by our CRS and this is a context bound if you haven't seen one of these it's a way of using type classes it just means that for this type there must be a CRS type definition available okay so now we know we've got the CRS type definition we can find a transform to some new CRS okay for which there is also a CRS time definition we know we've got this definition we know we've got this definition we can return a math transform this is all machinery so far we haven't seen the title work yet but what we know is that given this type in this type we can find it a mathematical transformation between them now we can do okay my co-ordinate is going to have some kind of CRS in it it's going to extend a geometry type of that CRS and down here we're going to have a method called transform the CRS of that of that coordinate now what this does it doesn't just do the calculation it gives back a new type when we call it so we say take this point that I've got in epsg 4 3 2 6 transform the type to epsg 3 2 6 1 5 so that's a new type in the system right when it does that it does the mathematical transform automatically we control that we make it do the transform before it gives it back now we have a coordinate in the different type in a different CRS and it's correct right why does this matter because now the compiler has your man all this machinery it seems like a lot of work but you only have to do it once people using your API now get to say I'm gonna show a map and I'm only gonna I'm only gonna accept coordinates in wgs84 or epsg for 3 to 6 right if I try and give it a PSD for 3 to 6 coordinate everything's fine if however I try and give it a coordinate in UTM zone 10 it's a compile error now it says you can't do that that's the wrong CRS so we just got rid of all our runtime CRS is right we don't have to do that check anymore the compiler now tells us you can't do that that's the wrong CRS so what we can do is for any CRS that doesn't conform we have this handy transform CRS we can transform into the right time it now compiles so we've taken this whole like group of runtime problems and turn them into compile time problems and now you have to fix it before it it will keep compile so you know it's correct ok so I am really racing through this but there's a flaw here the problem with the Scala type system which you very quickly learn is that it only exists in one place which is inside of the Scala compiler the rest of the world don't give a damn about the Scala type system ok so at some point you have to bring stuff in it and at compile time you can't see the future so you don't know what a CRS is gonna be if you'd load in some data it obviously has a CRS but unless you're very lucky you don't know that it's gonna be wgs84 it could be nad83 it could be some other CRS right so what we can do is introduce this idea of an existential type ok it's a type that there are some things I know I know there's going to be a type here first of all some CRS will call it there's some CRS there we know if it's going to be a definition because if it's a if it's a reasonable answer reasonable assumption that data you load in is going to have a conforming coordinate reference system so we can read that data and we can create a CRS type outfit all we need is the definition we can make that that CRS type we can put the definition in there tie them together so this some CRS and this CRS type need to be defined and this is your existential existential class this is some CRS that we know it's going to be there we know we have a definition for but we don't know what it is this is one of the coolest things about scholars type system the ability to put path-dependent types inside of other instances of things and have them be completely new types in the type system this is now this is a real type in the Scala compiler it's distinct from every other type in the Scala compiler so we just invented a type okay and we can use it it's not going to match anything else yet but we can transform it to whatever we need so what we can do down here then is you know we look up these CRS definitions however we do there's some identifiers in the file or quite often there's like an actual model definition in there we suck that in we create the companion type we create this we invent this new type and we say that's the type of this thing here's the definition for it and then you'll notice by the way the type and the and the value have the same name there's a reason for that what we can then do is import that type from the instance so we've read this thing we've created it we've got this little packet of information that includes the type and the definition for it if we import it if something comes in as a real thing like we just we're just not only invented this thing but we just sucked it into the Scala compiler and the definition that it needs the implicit that it needs came in with it because it's got the same name so then we can use this thing we can say well I don't know what this CRS is value1 it has the implicit that it needs so I can make coordinates in that I can then transform them to any other CRS in the system so now I'm reified that into some type that I know hopefully this is like making sense and I haven't just rushed ahead so fast that everyone's at this point lost but this is really cool not only that maybe you don't want to convert in you know I mean really a PSG three to six one five is just some other type in the system it means no more to us than this does so maybe we go the other way maybe we take the things that we have from other sources and we just convert them to some CRS that'll work as well we can go either way we can these are just types these are just types in the system all right they'll be time hopefully time for questions at the end I go very quickly over polygons but basically this is using the same thing transform CRS here goes down to the coordinates finds the transform for each Co ordinate maps over those and then creates a polygon in the new CRS so you see how this kind of composes and we can do transforms between all of these different things now what I think the thing that is worth mentioning here is that this code ends up running faster as well the compiler has checked this stuff so we don't have to do all these runtime checks anymore that's actually getting top speeding up the code we paid a higher compile time cost but we get faster and safer run time all right the result of this is that the API is simple and it's really hard to get wrong we create coordinates in some CRS if we give it the wrong CRS it immediately tells us in compile in fact the IDE actually tells us I was expecting a three to six on five you gave me a three to six one six that's not going to work you have to you have to rectify that and it doesn't take long for people to pick up on the oh yeah I've got the wrong CRS I'll just do a transform that'll work right okay this one I thought was worth mentioning the lat laws are angle based there they're like angles and asking for an area of an angle based CRS is kind of a weird thing to do the area will actually change depending on the latitude of that thing because the the pixel sizes change effectively so what we could do is introduce the idea of an angle based and maybe some kind of length or meter based CRS and we only allow areas to be calculated on this meter type so we would then say an area in square meters this depends on us having a meter CRS type definition and now we can't pass for three to six in because that will give an a compile-time error it says there's no evidence that this is a meter based CRS so you can refine it you can change the rules and make them more straight all right how am i doing for time horribly alright one last thing this existential idea it can be one see once you grasp it once you've got the hang of it can be used for a lot also one of the things that we end up working with is the raster data raster data rarely comes in wgs84 because it doesn't work well pixels aren't the same size so all of those UTM zones they're there what tend to be used for the raster we don't want to resample those or transform them because that's a very lossy operation with raster there's not really time to go into it but you have to like choose a new value for the pixels all the pixels are in different places so they end up introducing a lot of error when you do that so what we can do instead but we also don't necessarily know when we load this thing what CRS it's going to be in it depends where it is in the world so what we can do here's our grid coverage with a specific CRS we can create a grid coverage with CRS holder for that that has the existential type in it so now we say our rules are still here but now we're trying so it's easier to use right so now we say okay I'm gonna read in some grid coverage I don't know what the CRS is but I can tell you that it's going to be some type that I invent and here's the implicit for that type and now you can report that again where I use it which is the next one you can import that coverage coverage CRS and then you can use that as a type in your system so we we have our CRS loaded it's in three to six one five we have some coordinates that are in three to six month six this can really happen because these satellite images often overlap and edges of CRS is this won't compile but if we transform it to the coverage CRS that site it will compile and it will be in the correct CRS so we've again we've just saved yourself having a field in the middle of Lake Erie okay and this is just a kind of a refinement of that maybe we don't want to have to make you go transform this thing maybe we can just say I can crop to a polygon in any CRS I want so here's the polygon CRS has the coverage the RS transform that coverage that poly CRS to the coverage CRS and then one can crop it so you know it's just building up the abstraction layers and once you get the idea that there's no difference between the type that you define and the type that you just invent as long as it satisfies the implicit constraints this is all the same stuff right this could be a PSG three to six or five but it's not it's one that we just created and it has the information it needs to to use it all right I don't know if there's going to really be time to do this I'm just gonna skip straight to the chase here there is a idea that Chris Voigt and I don't think he's at the conference sadly I think it was Chris I'm like 95% certain it was Chris that showed me this many years ago and I promptly ignored it for about six seven years and then I came across it attributes again and I'm like oh you know a Titan that might be interesting for that so we get all these kind of loosely associated attributes on features on things in GIS data then you know like I yield like a crop yield for a field and a planting density for a field and they're all kind of numbers and things like that and they end up going into a map of string to any because you know we have no shame that is apparently what works and so we'll stick with it but we can do a lot better than that with Scala so if you if you use these type in next Maps you can do and some some fancy footwork with tight tags this code is all open source so you can go and look at it afterwards and kind of absorb it what you can achieve is something like this so you can create a team map okay team map of 10t map is sort of like if anyone's used h lists out of shapeless a t map can be thought of roughly as analogous to what an H versus H list is to a list a team a pairs to a map okay it's a map that has the type information tract in it like an H list has the type information tract in it so if we create a T map of ten this actually becomes a T map of int okay because that's what's in there that the ten is the value the key is the type which is ten if we add something to it this is just a plus method that I put and I put a string in there this widens out the definite or narrows the definition I guess it does an intersection type and it says this is now an int with a string because it's got both of those things in there if I had something else like an a list of int we end up with int with string with list of int this is how we do loose loose but statically typed attribute tracing in the libraries that we use so it yeah what you can do is things like this so I'm not going to go deeply into this kokis it's not time but what this lets you do is something along these lines here's a calculate yield at the top it says I'll take any T map as long as there's a flow and duration in there and I can produce a yield from that down here we've got a team app that has a field name with a flow with the duration because of covariance this is a subtype if that's a we can call calculate it and the result will be a yield we can add that back in and now we end up with this team map gets big the tank gets big if you do enough of this but that's the price you pay so you end up with field name with flow with duration with yield in it and we can't call this with something like feature with the yield two which is defined up there somewhere and that's because feature with yield two only has a field name it doesn't have a flow and a duration so you will get a compile time error when you try and do that okay this there's just not time for this I was going to go into why the the rich extension method but you'll have to know this this talk is available online so if you're curious as to that you can find out on the Scala days the rich the rich wrapper gets around a problem where you can't use equals : equals because the covariance rules are different from the invariance rules that equals KO na equals requires you can use a less than : less than but when you do that it infers nothing so if you introduce rich implicit the wrapped implicit it lets you freeze the type first and then go ahead and call the less than : less than version so it gets around a problem with type inference basically as there is the trick there okay so that's pretty much it I think I'm out of time but I will take any questions than anybody has this work was done for my company and I worked for called SIBO I'm full-time employee there as well as doing the escalating stuff we are sort of hiring so you give us a look we work a lot with Starla we do a lot of this kind of stuff that's Karma and my CEO role is training company called escalate software if you liked this check out our courses we will come and do them for you and there's udemy courses online as well and it will teach you the advanced courses will teach you all of this stuff and more alright that's it yeah yes when you go from one time to another where does the implicit come from good question let me I realized it was very rushed so let me see if I can find that oh so in that existential oh I have a lot of slides not too far existential here this is the implicit it's our responsibility when we create this thing to supply that implicit so up here you see that implicit vowel it's a pure it needs to be filled in we have to supply some kind of definition for it and that will be down here okay it is necessary to do a cast at this point I didn't I didn't point it out but as I mentioned casting is not your enemy you just need to know when to use it properly the scholar type system does not exist the outside of the Scala compiler therefore if you want to bring something from the outside world into the Scala type system you have to do a cast there will be a cast somewhere it may be hidden it may be cleverly abstracted out behind some other kind of method but ultimately there's some kind of cast going on because you can't invent a type for something that from the outside world without casting so this is where that see at that CRS is found it's you know here it's looked up down here we create the type and we set up the implicit and once we've got those two things that's a valid type with a companion with a tight class that we can import and when we import some CRS it brings the time and it's implicit into scope so they can be used just like any other one then so yeah that's really kind of the crux of the talk so I'm glad if that helped people understand any other questions yeah a huge fan of yeah I I like my color scheme it's available publicly it's actually based on one that bold radius did and I did some tweaking to I turn all of my files red for example so they look like errors that's one of the things I do it's actually not a compilation thing but it's like a little poke hey you're using a VAR you really shouldn't be so yeah that's available on I'll send an email to Alexi or something and people can get like be that because the slides and the code for this are on github and I'll if people like that color scheme I'll make that available as long I think I'm getting this I'm getting the knob that that's it so thank you anyway and I hope this is [Applause]