scala.bythebay.io: Flavio W Brasil, Dualistic Quotations in Quill
Recording: scala.bythebay.io: Flavio W Brasil, Dualistic Quotations in Quill
I you so yeah hello and my name is flora Brazil like he said and basically today I'm going to talk about quill and how it works internally so high this thing's at compile time I imagined some of you have heard about coil already but i will give like an overview of the library and what it does as well so when my i work here at twitter as a software engineer i work on the core services team and more specifically i work with the debates microservice everything scallop for a few years and i really love open source so I'm always working on something and the last thing that I've been working on is the coil library and it is a compile-time language integrated chord so the idea is when we are talking to services like databases or other systems sometimes we have a target language for instance single queries are graphic UL queries and it's not so nice to have to write another language within your host language so cool is a library that allows you to express queries in the host language so leveraging scholars syntax and also the type system and then execute these queries against a target language so just to give you an example this is a very simple query select from exact name and the difference of age between two people a and B select from the couple stable people a people be and then join by the couple stable little persons to people and then find all couples where a is older than be okay and then return the difference of age so it's very simple and there's a way of expressing something very similar to this in scholar that is using for comprehension so we can say like that couples and people are tables in memory and then we use a fortification we can get one couple one person a person be applied the filtering in the for comprehension and then use the final value okay but this actually has a problem right we are if we have everything in memory we have to load everything from the database in memory and sometimes it's not efficient so sometimes it makes sense to load everything in memory but the general case is not that so how can we we would like to write a query like this because it's a single language we are using scala but we don't want to write it because it's not efficient so what can we do about that turns out that someone has answered this question and the oh okay so if you see here it's actually very similar so turns out that someone has answered this question and Philip water is a very important person for the functional programming community so he worked with things like defining moan as defined in the type classes and etc so someone that we should know how most of them of us are using things that hit a fine and one of his latest work is this paper that is a practical theory of language integrated query so it's a very interesting paper there are many talks about this that he gave in multiple events and it basically has like to answer how can we write this kind of querying the hole in the host language and then execute against a target language so it shows it gives two main techniques to support this kind of language interpretive query the first one is quotation so this is a way of building a DSL the quotation approach and instead of having this is normal scholar code that will be would be executed at runtime we basically quote this block of code so it's a method call in this case oil implements this method called as a macro and what happens here instead of this being called that will be actually executed at runtime this code becomes a forestry so the macro receives a parse tree from this color compiler and basically quill will parse the tree from Scala and then generate an internal aest that is more or less this it's not readable here but it's actually a very simple ase only with no operations and things that we can translate to the target language i will show you an example where we can read with the SD and the second thing that the paper presents is normalization rules so it turns out that building the dsl is the easy part the hard part is from realizing things and then generating generating the final sequence tightness and the paper brings like this normalization approach that's more or less this don't be scared it's integrated something complex or something like that but it's actually very simple if you see this transformation being implemented in mark soft link you or even is league in Scala it's like thousands and thousands of lines of code and this is very simple and this is very elegant it's like less than a thousand lines of code in Scala drink amenities and it's much more robust than normally the other solutions are and ok I won't be able to show all the organization rules but I will show one of them to you today that is a far far well this would this is one of the most important ones to generate sequel queries so the left-hand side is the pattern that we should be looking for and the right-hand side that is the second line is dead normalized form of that pattern so instead we it's hard to understand it because it's using another notation I think this is f sharp and it's not possible to translate these directly to fort intentions in Scala the way that we can translate these three Scala is using two flat net calls so it's being flat map to Q and then flat map to our and the normalized form of this asian is B flat map to Q but the second flat map becomes part of the body of the first flat map okay so there is just like a parenthesis change and it's interesting because of paper doesn't doesn't paper doesn't mention this we can do this transformation because there is something in mobile ads that is the associativity law that means that says that both versions of the composition have the same meaning but for our power transformation to sequel the second one is easier and it's a step towards the final sickly statement and we have the ASV produced by via the quotation approach we have the normalization engine by based on the normalization rules and also all other reduction phases and from using these two steps it's actually very easy to generate the final sequel statement that is as exactly the same as the one that we were trying to reproduce in Scala so that's the overview of the library do you have any questions about this question I recognize what which one here sorry panting Oh entity yeah so far quill the entities don't need to be anything else they can be just like a case class they are just a case class and the parsing will see right there is this type here that's been queried and it will use it to say this is a source table you know the questions cool so i will show you now one example of a more complex query this is from the t PCH benchmark it's a very well-known database benchmark and this is a korean number two so it's this big query it's not so hard to understand like it's just big and we can use the same approach and just translate it to for confirmation and it's more or less the same thing right you have the same structure but with the quotations and the way that cooled this normalization and the paper also defines the normalization we can do something better than this we can extract small parts of the quotation and then compose small functions so here i have this part of the worry that is actually this equal to this other part so what can we do we just extract a function right so this is looking for the availability of a part given a region so selecting from the relations between parts and suppliers and then selecting from supplier patient region doing the filtering and then filtering by the region in the part and then using the final value the final regular of disco tation and then we can reuse this quotation to define another small quotation that is the minimum cost and given a region and apart and find the availability of the parts in the region and then find get the minimum cost for that part in the region and the last one is fine part this is actually very simple given the size and a type of part query the park table and filtering by the condition and here we have the final query that is much more readable with easier to understand because we have like these small parts that we are composing so find a part and then find the availability and select the one that has the minimum cost in the region right so this is very powerful and basically the quotation gives you the tools that we have for normal normal Scala code like composing functions using higher order functions and things like that because the normalization engine will apply beta reduction and other reduction phases to normalize everything and the final sequel statement will not have the functions so we can abstract using functions but the final cyclic statement will not functions any questions if you have questions you can just stop me okay and as you can see here is much more readable and it's very nice because basically you can reuse the fine part a quotation to do other queries you can reuse these small functions to do other things as well but this approach so this is how I implemented the paper and this actually has two problems so first normalizing the queries and generating the final six to eight statement is actually somewhat expensive it's not for free you have a price to pay and you also have a long feedback cycle when you are creating the occultation it's hard to know if the quotation produce the Prairie that we are expecting or the database will be able to execute the query so I started thinking okay this paper is great could I do something to make it better I could I make it in a way that it's the feedback loop is Charter and also I don't have a runtime overhead and I thought okay i have already i'm already using macros to do the quotation and i ended up creating this technique that i call Jody state quotations okay so this is basically explaining explaining how oil works and do things at compile time and those things at compile time so here i have it on another example we have two K's classes and then i have the quotation person by name filtering all people with the name that's passed and then i have a second quotation that is contacted by person named that's using the first one okay and then finding all contacts of that person so what happens here when I'm using the first quotation when I'm calling the quote method is a macro methods and I have all the information about the quotation I have the AST I have everything and I could just translate that to sequel statement right but there's a problem when I have a second quotation and I have a reference to the first one that is person by name I [Music] don't have any information about that quotation basically the macro system will give me just an identifier saying okay there's a reference to a person by name here and I basically I don't have any information to continue and generate the finals excitement because I don't know what's there and the technique that I used to implement this is basically this when there is the quote method it's a macro methods and this transformed version the output of the macro that is an instance of the quoted type the coated type has the type of decoration and then it has a method that returns the aest so as you can see here the AC is very simple it's basically a function I'll show you and yeah i'll show you again and then this gives me access to the quotation AST at runtime but at compile time when i'm explaining the second quotation I don't have access to this method I don't have the information pile time and what I did was create the map and an annotation to the method so with this annotation that is that has a funk the AST and basically scholar annotations are part of the time signature in scala and i'm creating a type refinement that has annotation and once i have a reference to that quotation i also have the access to that to the type of the quotation and i can access the quotation as a compile-time value ok and i also have to create a method ID because scholars and has some bugs with unifying types we find type refinements with annotations and basically what happens when I have the call person by name I'm parsing the quotation and I have all information about the current quotation that is contact Swiper contacts by personally but I also have access to the aft of the person by name function and I can splice the ASC locally and this basically propagates to the point where we actually call the quotation and execute the query in the query is generates the sequel statement ok any questions about this part too much is is it possible do you understand why okay nice and then but this approach has some limitations that's important to know well the first one is dynamic queries so if the query has the structure of the query changes depending on runtime values not not for instance it's possible to have bind variables that are run time values and they are fine they don't change the structure of the query but there are types of queries where the structure changes so let me show you an example here so let's say that the person by name rotation checks if the user is admin and then return all people if it is not add me if we return only people that are not protected like a visibility filled ring and what will happen here scholar will have to infer a tight for the first rotation and it will be coated this upstream to the entity query and we basically lose the type annotation right so far the NEM quotation and will doesn't have the information to continue doing things at compile time so that's the first limitation and the second one is okay and the second one is when we the user does type widening so let's say that we have a person by name but the user gives I explicit type to the quotation saying like quoted from string to query so this means that we don't have the annotation anymore we only have the type coated and we don't know anything about this first quotation and we won't be able to generate the query at compile time okay so that's a limitation but that's not cool and but there's something very interesting and I'm looking forward to be able to migrate you Scala matter that is we will be able to get rid of the second limitation so we could we could give explicit types to cart Asians and so it would be the same thing you could use the type widening you could say like this is a quoted from stranger query but there's this modifier here that is in line and Scala meta will instead of considering the first quotation as normal color code it will spice the tree the original tree into the expansion of the second quotation so I will have access to the original tree of the AFC setup just an identifier and this is very interesting it would be possible even to support somehow like and I nen co tations if we could graduate scholar code and compile time but I'm not sure that it's totally possible to do so I will play with it when I my way to choose collimator and this is awesome and it's very we have who's it just doesn't oh yeah that's a good question if we don't have okay he asked what happens when we all lose as a type annotation it it will fall back to runtime query generation ok so if oil doesn't have information at compile time about the quotation instead of applying the realization rules or generation at compile time it will fall back to run time for generation yeah and it's interesting because normally people think like quills a lot of like macros or something like that it's actually like most of the code is about generating queries and its normal color code that's used or compile time if the query is known at compile time the AST if not it just haul bags to the same execution blood at runtime you know the questions restrictions in terms of operations that you can do or yeah that's true so yes okay yes and what kinds of restrictions we have when we are the final tations right and we have to so the parser will be able to parse only things that are supported by the target language so we will fail Chi to compile if there is something that's not supported and you can do like high order functions you can do function calls you can do most of the things that you can do is color but they need to be operations that are supported by the target language or the question okay and it's interesting because if you think about the quotation and how it influences the way that the code will be executed it's actually when we are defining a quotation that is static we are doing like metaprogramming the user is doing because it's affecting how the code will be actually compiled so i have this example here so like I said entities can be are basically case classes normal case classes classes there is no mapping cold there is nothing else it's just a case class and then we can define a quotation that is growing all people and have the name John and then map the result to the age of the person and then please pay attention here so at compile time we are going to get the annotation from the quotation and then trans do the normalization work and then generate the final sickly statement so this means that we have very low overhead at runtime there's some overhead because we will also does encoding and decoding of values and provide some abstractions or Java wipe resorts leaking and things like that but it's very low overhead the query generation doesn't have any overhead at real time and the feedback cycle is like awesome because you can see at compile time we would like compile or we will be valid or like is it what I'm actually expecting and you say like a normal contains operation yes yeah it's possible so that they we don't have like support for all sequel features but most of them are supported so for instance if you have a set you can use contains if it will be translated to in in clauses in Mexico for instance okay so sorry like half of that so can you dynamically at runtime the values for for example of John would eat appetite I see ya so he asked if we can address at runtime splices something like insert something that is a runtime value and yes so there is a matter that's called lift and it allows you to lift a real time value to the quotation and in sequel it will be a bind variable and with health problems with sequel injection or something like that another questionnaire so that's a good question right now we don't have any tools or details or generating databases and we don't have any like management about versioning of databases or something like that we have plans to do that but we have a feature i will show you it now that is compile time corey probing so i can for instance change the person name age tune a to H E and the completion will fail so I configure my compilation unit to have a database connection information for database connection and will will at compile time talk to the database try to prepare the query and in this case like the column H E is not neither database and the query will fail to compile checking yeah that's a good question we don't have like numbers on it and you can expect more time for completion because you have you are doing more work but the Scala compiler is already very well so it's possible that the time that spent like talking to the database is much faster than like everything else it's color competitors yeah that's a good question optimization these are the entities available it should have these views because I already waiting like yeah this is something that we have to think more about and I think we don't have all of the user what users are not using this feature yet so if we start having like bigger codebase we have to think maybe about having like okay now I'm going to release so I'm going to compile it with the database connection and verify the queries but for now normal development development I could just disable the query probing or something like that but you know this is something that it's more as an open question have to experiment with it another questionnaire good question so it oh sorry yeah so he asked if the if the case classes need to map directly to database columns and the second question is how we deal with more than twenty two columns okay so the first question and when when we are defining the case class we can define using a identifier and then there is a way of defining a quotation that says for this call calm use this other nay and then you can build all the quotations on top of that quotation and there's also something that's new and that is the matter instances that quill has that allows you to control the way that will expand case classes that is another way of maybe renaming things and for the second question in case classes spark more than should thank you elements right they don't have an apply methods I think in two to 11 and so coil doesn't have any limitations in terms of number of columns so you can use as much as you want my yeah it's something that you have to it's more an operation question right you have to choose when and how you're going to validate your queries right if you have a stable environment and the staging also has the most updated version that will be executing in production you could like always have the completion worn like the compile time curry probing enables if not you have to do something else maybe enable some in some environments and others not and this should give you an overview of what we have now so there is we have a few modules so there is oil car that has a quotation implementation organization engine and all the macros and this car is agnostic to the target language or drivers or anything like that so we could extend this core to generate a graphic UL queries spark SQL queries we could generate even queries on top of scholars scholar collections in memory in a more efficient way so it's very extendable extensible and if you want to implement a new extension feel free to come and talk to us we are going to give you support you generate a new to create a new extension of the car and then there is squeal SQL that has the driver agnostic SQL generation it doesn't know anything about the actual database driver and then there is oil hey sink Mexico and post-race they provide a fully non-blocking i/o for my cycling post agrees this is not just a wrapper on top of blocking jdbc there's also integration with enable my sequel infinity poster is that is based on finagle also fully non-blocking and that we also have integration with jdbc if your if you don't have like a very high scale it's fine to use jdbc but if you have to scale your applications better if you could use non-working i also use the other modules and we also have a new module that is the coil cassandra well and that generates another target language that generates eql using the data stocks datastax driver and it also has integration with monex for reactive streams and we are planning to extend the reactive streams integration with to the other modules as well and that's all I have ok and you can follow me on twitter and github and we are hiring we are looking for maintained errs we don't pay any cash so you are a of that but my team here twitter is also hiring so if you are interested we also very Hardy and yeah that's it any other questions not my services I don't use relational databases okay so yes if we are using oil at Twitter there are some teams teams using I don't use it because i don't use relational databases in my service that's a good question so I think Rob created a module that integrates with and quill but it's not updated I think it doesn't work anymore but maybe we can also integrate again so he has he's asking about the differences between quill and slick and so the way that the dsl is field is very different so a slick uses an embedded dsl so it's something that will be executed at runtime and quill uses the quotation the SL that will execute will not execute anything at one time so to produce the ASC the query generation is similar but the implementation that zig has is not so robust I think because of it doesn't have like a basis like a paper this as this one and what else but like it's the same space the same abstraction as they think other questions okay up one last question sorry what exactly is this module for my seagull possessing so he's asking what is the module for a sink and post-race and my sickle it's a non-blocking i/o database driver in Scala for my cycling pop quiz so it's not something that we implement it it's not part of the coil project it's a another driver cool think that's it thank you [Applause] you