Scale By The Bay 2018 Unconference: Rob Norris

Scale By The Bay 2018 Unconference: Rob Norris

Recording: Scale By The Bay 2018 Unconference: Rob Norris

it's all type driven we say I'm going query string and there's an instance down there that says okay my decoder for my private plane my decoding logic for Strings ultimately the problem of that is there are other things on the schema side and it could be a blank header could be a text there are a lot of things so and so I wouldn't be how much me explicit there there are a lot of ways to map a scala type to database and you might not even do it consistently within the same database you might have this value and over here in this table we're doing it this way never in this table this way and for a given data type so so on the schema side there's a lot variations on the scala side a given data type might would have to have those different mappings so it's not it's not a I promise I don't think it can be typed ripping it's this the the the D sterilizer is not unique prototype unless you start doing all this new target so what I found is that when I've got a big model and a big schema using do B I'm having to do a lot of new typing in order to enable the automatic derivations to give me an instance that I've known and I thought I would try something different just like it explicit and Treves data treated more like parser Combinator's less like Mon Weitzman functions and just see how that worked down in the design I'm not wedded to it the way I'm doing right now do you use refined types for like specific cases of 16 ranks streaming close race no and that's something that I've thought about doing because and you when you get the row descriptions back it will tell you it'll tell you this is a Viper you know a white padded char that's you know that's that's eight chars long for instance so yeah you couldn't you could have her fine types associated with this I didn't want to pull in okay so let's let's um let's move so this is a similar one but it does more interesting so the session business is the same here we have a case - as a country as a code it has a name and as population so I'm creating a decoder for the country so there are encoders and decoders and then there's a subtype of those called codec that's that's for symmetric matter so you can you can just define whatever whatever you want so codec can be you can forget that it's a codec industry of those encoder as a decoder so these things bar chart bar chart and a these things are actually codecs but we're just cannot codecs decompose with each other to do sort of like incremental migrations of a codec from one version to another so like if you have a codec that's like version 1 can you create the depth of the codec for version 2 and then combine them to get like version 2 depends on version once you just describe the dip and other codecs change over time and then so a migration of your codecs becomes just providing incremental changes they added on top of each other potentially I may not understand your question - so the cutting the codecs are invariant semigroup whole functors so you can certainly you can certainly take the an original one and map out in both directions to to evolve the mapping [Music] okay so maybe we can talk about I think I'm okay so let's let's look at this one so we've got so what we're doing is the the the twiddle there is for as we're creating a product so it's the applicative product so it's creating a left-leaning ageless basically so when you so when you compose encoders that way so include a twiddle encoder B gives you an encoder of H little B and H little B is just another way of writing the parity okay so this does not apply there it lets you unup lie what is a left-leaning list appears disservice and then your thing is just syntax okay all this chameleons with the pairs so what I'm doing is I'm just reading bar chart bar chart in eighth and Scala types for those are string string so that's so now I have a decoder for country and then in my query here so I have a few things going on so here I have an argument it's a bar chart so that's a pattern that I'm going to pass and then it's a query that remove the returns countries so now what I'm gonna do is I'm going to use the extended query protocol which I need to use because I misses parameters so have a session elected to prepare this statement okay and that gives me another resource the statement has to be closed after it's after we're done okay doesn't have to be when you close the session and we'll get cleaned up but in from a structural point of view it makes sense to to make this to make the scope of it explicit that's a design decision so we have a prepared query and then the prepared query there are lots of things you can do it you can get the monies back as a list or whatever but I'm not streaming so this isn't so I passed the argument there just a pattern so there's any countries that start with a and then there's the Trump side so it's going to read chunks of eight rows over time okay so this is an FS to string okay it's a stream of i/o country so avala map just takes that it says taking gentlement and to effect so it's gonna print it out then compiled rain says okay just take this whole stream program and turn it into a single audio action you can see you can see what's going on so so it parses the statement and it gets a success back we asked for a description of it and it gives us a good description of our parameters so that's the 10:43 is the weight of that type so I have a table where I can work and then a row description that tells me about the columns of the kernel and this is just diagnostic right now so it turns out that I have said that it should return of our territory a day but actually is returned by paddock HR by Charlie okay this will turn into an error so this is going to turn into a zombie throws an exception in do B you can write unit tests that do this checking would sort of and sort of do it out of ban but in any system you're writing you're going to have tests that actually run this query against your duties at least hi I just want to I just want to go ahead and perform this check every time there's actually a parameter that you pass to the session so always check them with each other so in development you can just say as you're developing it'll just make sure that your types are all aligned and then in production if you wanna check that's the way I'm doing it so there's not this sort of extra okay so now so we're binding mining this argument to someone called a forest it's like a cursor and then we execute that cursor and I want eight rows back so I'm going to get these eight rows and then we omit that a chunk of rows toaster and that portal is suspended which is there more results we do it again well internally yeah ii thought use that does this internally there's portal double use and that's what the stream is doing so the the fs2 stream has resource safety in it too so what it's doing is is piggybacking on the portal resource up and so with the stream terminates under getting admission so the stream throws an exception or something then the stream will take care so there are three resources that can invest certain assets with demo set that you were sometimes yeah your father your assumption and we considered pushing pulling the assumption back and doing it at least making a list of assumptions upfront you can sew the yeah you did that up to a point but a lot of the time so if you have me if your queries are all our literals then you can do it a compile-time I mean the example macro so you can do that and you can do it as sort of beforehand like you're saying but in practice what happens is you end up you have some queries like this that are sort of all there but it's very common to construct a query from all these little fragments that you put together so you might have two queries that differ only by the foil eyes and you can factor that difference out so at compile time you don't know the what the state is going to be and then even at run time depending on the arguments that come in you might construct different things so I mean I would push that to a test case probably but but maybe you could have a sanity check is that sometimes your production database no I think it's amateur okay so are there any questions about this or about how any of this stuff works so how much analysis is done that would be very for example if you have a median location - this was absolutely yes the whole can final and you get a and you'll get a so if you like if we if we ask for a name again I think what will happen is it will work the analysis but you can still pull this through collars out so so I want that I want that continue exception of these times the short answer is that I don't I don't look at the secret so look at the grammar like deep does he find your tables in Moscow straight away and then you just analyze how the structure looks like that's what it slavery does you can analyze everything right how is the structure of your own database on all the tables right Polly date then time time like all these analysis that is running says front end right this is all run until you ask okay so let's let's go on let's go into the next one so these are just minimal careers I went on it so just kind of sketch you might constructed my we're going math and I'm just gonna this is going to be a now mouth I'm gonna implement it I have a post grad student at Boise so what I can do is given a session for any F that has this bracket behavior this is required by those required resource that kinda does safe safe bracket requires that history so I can say all right when we asked to add these numbers together I'll prepare this at statement and run it and expect you need answered by the same thing for doing the square root and now I can get the session I can turn it into an these math things and now I have this math object it's not really a database table with it it's just something that knows how to do math so this is you know this is this is kind of a way you can build up on top sort of sort of and so where you can see what it's doing it's it's it's just doing what we were doing but a problem potential problem here is that it's doing this work twice it's preparing this ad query 2 times because they're using it twice which is a little bit inefficient so what we can do is something like this so we'll just change our from session here to go ahead and prepare these statements before okay so repair these statements and then just return return math that just it just uses these things that are going to prepare and run this one we only get to perc errors and they have been upfront and then all the actual application work takes advantage of that and because this all happens inside a resource everything gets cleaned up in the right order so we're not worried about making deprecated State so depending on what you're doing that can be useful preparing States this is something that's hard to do in to be and it's hard to do we can't BCE and make sure everything gets cleaned up right so I think this is that's this resource gives you a really nice way to have these things these these resources that have clean up code associated with them and compose them together to build bigger bigger resources so there any questions about this I'm not quite about this session one single does that imply there is or session pool yeah there's a second pool I'm not gonna show it to you but if there's a second pool and it I had to create a thing called leak that will be a resource and that's cleanup routine and so we've got a pool of deep resources and the semaphore and I'll show you a similar problem what's that okay yeah so this it's actually generic it'll work within your resource okay yeah there is a rule if it works exactly it looks exactly the same okay let's look okay so Postgres has it's not just a request response model and you're talking to the server the server will actually tell you things asynchronously these asynchronous messages will kind of get in early whatever you in practice they only arrive at certain points in the protocol but in principle be ready for so some things like like if you change the change of time zone you will get an async notification time sensors my choice would need to prefer if the character encoding changes but but at least it tells you right or decision but one of the really cool things that most guys does that very few people as far as I know have exploited for good or evil is it has this notion of a channel so you can say I want to listen on each other and you can say it's and later on you can say I want to send a message to this channel so well look at this show so what we're doing is we're getting the chat a channel from this session this ID constructor just says this thing is a valid construct it's a compile time guarantee that this is not like it seems that way we want to listen with a buffer size important to assembly about the buffer size well take the first three messages and print them out okay so if we run this so it's run this query listens to and that we connect so I can say notify so this is this is the Postgres shell different program when I do that the notification immediately shows up okay so let's start transaction now the message goes out to their transaction okay so you can have you can for example set up a rule on a table this is something anybody inserts a row into this table send a message to this channel that tells you that their reports the primary key that was generated on this episode and that message will go out but it's not going to get sent until the transaction is committed so it's as if the message was never sent transactions which is really cool right so this is that problem in my life for a long time with with crud apps because you got an app they can't catch anything because you know no one's stuff is getting changed out from underneath you but 9/10 database tell you when stuff changes which is really cool I've never seen anybody that advantage of it it's a way to make these apps you're not you're not pulling you we're just okay so one one last thing I want to do [Music] question is to topic the channel is not a topic when you listen yes but the notification so that so under the cover who has them yeah under the covers these notifications come in and they do get posted to a channel that this one channel all listen to us is a subscribe to that channel and gets the street so if nobody's there listening them so you have a lot of people listen yes they're notified you can say channel okay so um another problem with do be that bothers me it bothers everybody because this is a general problem functional programming in Scala that's if something goes wrong you get stack trace and stack trace is completely useless because what you see is one line at the point where the exception was raised and then you got like 50 lines of Io run loop this is kind of done suspend or whatever and and it's just it shows you the state of the runtime it doesn't show you the logical state of your program so I wanted to improve them to error messages this is something I've just been working on it's kind of it's not quite not quite done but I think it's good so here's here's something we've written it might be writing them be wrong so but stuff happens okay so here we okay so we read the sequin section this is not a Java sequel acceptance right so this column doesn't exist and we look at this it's just fantastically useful but if we scroll up the message associated with the stack trace is useful okay so it says this is the Postgres theorem this is where in the Postgres source code that error is defined here's the push press gives you a description of the problem it gives you a hint sometimes I'm so here at suggests what column I might have wanted Postgres tells me the position in the in the statement were occurred so I can get a carrot there and take it here it was been um I can tell you here's what was happening on the low-level message channel before this exception occurred so this gives us this gives us an idea of what was going on something I didn't show you is you can use the stuff concurrently so you can have like us you can be reading two streams at the same time the chunks will be read in an interleaved fashion and maybe there's something weird going on that caused an exception here due to something happening concurrently that that might give you some insight of what was happening and then there's a decoding with the error response itself something else I want to add is the source location of the query that's easy to do it and so I am thinking about is having requiring me to run this in a like a wisely and has an environment around what you build up a certain logical runtime stack so you can put markers in your code saying here I'm doing this here I'm doing this room doing this and gets her a logical trace back just starts so you can have sir that's an idea at the application level what what am I actually trying to accomplish at this point to give you some more insight now I'll show you a few more examples this is a different error does this operator doesn't exist because there's something wrong with the work the characters so so we're we're saying this thing is a large yeah so now what happened okay so now we've prepared the statement right but now we're supplying arguments to it and something else coming on so in develop syntax for integer okay so now now that we've actually called with arguments it can tell you what the types and the values of these arguments are okay so when we come out a little bit more my more insight into what we did wrong here and again the message this is a higher level error that's that's happening and here this is I'm you've been experimenting with passing with noting line knows as part of the API isn't how it is so there's a problem related to use of unique at this line right here so and those where we called used unique did you mean to use option this so I was you know inspired by the posted terrorism so I feel like I feel like kind of the bar it's been set for this and like you know like a few years ago everybody was like I wanted we're gonna type check documentation that's gonna be like the big push for usability for stuff and I didn't ever messages trying to get hairs representative data structure where if I needed to like master know right now the way this works is that the two string on the mask you to string on the exception itself is is this thing next this is just this whole thing is this tax base is being printed out with just the description is very allowed so there's likely a better way to do it that's kind of what I'm doing - of this I mean you look what you what you might want is some data structure that you can you know need to right now the tool that can just kind of as a first pass just showing something to the user that that's definitely inspired by use your build is okay that's that's all I've got I'm happy to answer questions how do you do like retreads so you would do that in terms of you do that in terms of just goes down that I started standing me that's a x-files you can you can attract traffic exceptions you can do the logic explicitly so for example you might get a magnetic devil that's the kind of thing that happens and the solution to that is to try again yeah yeah that gives you what if it's something that makes sense I appreciate everybody listen I really do because yeah yeah so I mean it's a doobie dubious useful made a lot of people post a lot of you give you weird databases like like I mean like databases that are just just like like it's just it's just barely implemented ABC and it's nice to be able to write reasonable programs you know people using the Oracle which is like it's a very very cruel piece of software to use want to change recommend Minister solid and Douglas I'm probably going to change these battle style which will solve solve some expressiveness problems and add some new problems because if you either leave transactional and non-fans [Music] I'm using Kodak which is Michael's library for binary serialization on civilization so it's just talking to protocol directly so their latest air socket and then the socket speaks in bit vectors which is sort of the primitive for this codec and then messages on top of that and then a version of messages that takes these asynchronous ones and throws and shuts them off to a channel and then another layer that speaks in terms of logical exchanges of messages which have to be which have to hold on to a semaphore we can't those exchanges and then on top of that since the session it was so there's like five layers so you cut out the middleman yes so okay so it does a few things one thing it does is you can give it you can give it a decoder so opt just says this so that's one thing it does so let's bring a little tricky part for encoders here if you say [Music] and that will interpolate a literal yeah so so that that interpreted the little the literal value population that something do and do I always everybody another thing you can do is you can do that okay so this can be another fragment applause interpolated and then those parameters would get inserted into the right place into this big type of those types in do be those texture existential you can't see a fragment is just compare object that closes over the argument itself these fragments are parameterised by the argument their their parameterize by the area from the argument the types of the arguments you give them later so you delaying specifying the arguments to this so when you get back is more like a function yes great thanks