Scala at Sungevity, Denotational Semantics and Play, SF Scala @Sungevity 20150224

Scala at Sungevity, Denotational Semantics and Play, SF Scala @Sungevity 20150224

Recording: Scala at Sungevity, Denotational Semantics and Play, SF Scala @Sungevity 20150224

hey everyone welcome to uh the nerf center of clean energy in oakland um first of all i want to say a big thank you to uh kachi and the uh sanjava tit team for for making this happen thanks very much and also thank you for sf scholar thank you all of you for coming out to uh to see us it's appreciated so let's just kick this off oh really okay cool so the first question i typically get asked when i do a phone screen is you're right scala is it for the panels and the simple answer is kinda yes sort of no so what do we do here well we write software um we're just wrapping up a major uh second iteration of our customer experience uh application uh we're very proud of building um probably one of the one of i don't know maybe less than a dozen siren hypermedia apis we're very keen on the acs concept so we've built out a pretty sophisticated api that exposes our rich domain model we also do cool stuff with monitoring so we're actually monitoring customers pv systems and um kind of aggregating information so there's a definite on-ramp for us there in the future for things like arca and particularly spark so what do we do one of the big things we built out with our api and also our client applications is we we actually handle interactive quoting for pv systems so it's a big deal for us to explain the value proposition of what solar is in terms of here are the financial options for the various kind of products that we offer and here is the environmental payback it's very important for us to also be a trust-based business so we use our scholar software to really build a relationship with our customers over the lifetime they have pv systems the thing that we're really excited about that we're going to move on to next is our next generation of design tooling so we're looking at where scala fits in for very large scale automation of uh designing pv systems using rooftop imagery so this is our stack right now we're big into type safe products we use play we use slick a lot of scala tests we're also moving into reactive and of course we use sbt for the future we're looking at akka and spark and figuring out how we can do some how we can move into the big data space with all the data we're starting to collect so i'm going to uh i'm going to hand you over to luke smith who's going to talk about functional domain modelling and our experiences with it um so thank you all for coming and enjoying this growing company thank you yes so i'm going to talk about how we do functional domain modeling here at syngevity and the subtitle is beyond buzzword compliance to because my objective is to explain that uh we aren't simply putting the word functional in from domain modeling what is functional domain modeling it's making functional code correspond to real meaning in a language that programmers share with other people including uh other people in the company and customers why would we do this well sungevity needs to tell a very simple story but a true story to each customer about their specific interaction with a very complicated problem which is getting solar on their roof customers don't need or want to know about the complexity of the various jurisdictions the funding products the taxes utilities funding providers and all kinds of things that they don't need to know or care about so we want to tell for every customer the simple story of solar for universal need or it's fun and that's only the first bad acronym i'm going to introduce in this talk but basically the simple story of our business is singevi looks at your home designs a system estimates how much that system will cost and how much electricity is going to produce and then shows you the resulting savings and then we put it on your roof and we give you access to that installation's performance and design information over time telling the simple story successfully and reliably at scale means rigorously accounting for a lot of complexity and through domain modeling it's an idea that's commonly associated with traditional stateful object-oriented environments but not tied to them it's fully compatible with and maybe even more suited to functional approaches in using scala but how is that well my assertion is that actually functional domain modeling is the original domain modeling the word domain comes from modeling semantic domains uh in linguistic parlance a semantic domain is a set of concepts that are grouped by having something in common so like the real set of music notes types of snow pizza toppings etc and in computer science in reference to programs a semantic domain is a mathematical object for rigorously assigning meaning to programs using denotational semantics relating the program to the actual set of real objects and it's compositional it does so by figuring out what the various parts of the program mean and how they're composed and then figuring out what that means uh dana scott along with scott's someone named strakey was working on this in the 70s and it's a pretty old concept lambda calculus at the time didn't have a mathematical model relating it to functions so when he was writing denotations for programs he then needed to write create a model for relating those lambda calculus denotations to semantic domains so scott domains are more restricted than just sets and they provide a general semantic model for computer computable recursive programs but most cases we don't have to worry about that and we can think of the domain as a simple set and that's true in this talk too so long story short scala and lisp you know adopted syntax similar to these semantic denotations on purpose and um you know i think that we can i can assert that functional domain modeling is the og domain modeling um it's useful because it builds up compositional denotations and it's very um compatible with functionalism because in fact it has functional origins and so even if you're not really interested in building up formal mathematical proofs it's still a good way to communicate about your code base and its real structure so i'm going to talk about a a toy version of the century domain called utility consumption sites why not just houses well because the thing on the left is a utility consumption site probably but the thing on the right is not so cengevity is mostly interested in places where we can offset someone's utility consumption by putting a system on their roof and we don't sell off the grid systems yet so utility consumption sites so back to the story of spun um this value proposition this core product that cengevity provides we can identify components of it that are semantic domains sets of real world objects so giving them names singevi looks at your utility consumption site creates a system design calculates the system price and your production estimate gives you a quote that shows your savings once the system is built we provide access to your installation showing the system design utility consumption site information and now your utility consumption site is also a solar production site that provides some kind of monitoring access and arranging these according to their sort of logical overlap if we just consider utility consumption sites well a subset of those are utility consumption sites for which we have designed systems at any given time a subset of those are priced a subset of them also have production estimates some of them are priced and estimated and then once you've got pricing and estimation you can have savings right and the journey from a to b is essentially giving someone a quote um for the purposes of this talk the operations are really simple but assume that these are things happening on different systems over different periods of time could be expensive calculations etc and finally c to d is a journey from quote to installation where we add the solar production site so we can imagine you know we're building an api with some routes that are going to carry us across these state transitions in these overlapping sets so we can visualize this a different way which is as a grammar of the domains and we can think about which products of those domains that we're interested in so installation is a product of the solar production site the utility consumption site and the system design the quote is the a product of everything except solar production site and then finally what's at the top what what actually describes the entire story well i've called it the spunit sorry we love that acronym here anyway another way of fully labeling this graph as products is like this and these emphasize these are the real world sets right the set of all possible utility consumption sites and the set of all possible system designs is a domain so we can also imagine it as a state machine where we're trying to move through these states and as you can see we have some options right and this shows that there's an opportunity for parallelism and at least some of these operations and if we you know imagine this as a play up where we're defining routes we can actually say well those transitions the movement through this model can be defined by routes like so and so when you post a system design to someone's some existing utility consumption site you could carry we carry it through adding the system design composing it pricing and estimating composing the results of those calculations then calculating savings and finally arriving in quote so this is a state machine that performs a rough denotation of what the route would do same here with posting a solar production site to move from quote to installation so doesn't include everything in quote so we're actually forgetting stuff here and you could look at this as um a moment where we actually achieved this unit right we actually achieved the maximal production and we could log it or something but in this model i just said well no we're going straight to installation but the point is we've now at this point traversed a path that includes every element so how would we actually implement these kinds of things how is it a guide to actually writing code finally there's some scala on the screen well if we define primitives like a utility consumption site about someone's address their usage information and their utility price and then a system design as being this this domain which is just the product of those primitive domains right we can also define all the other objects all the other sets as scala and scala code and some of them are traits and some of them are concrete just to make them composable but basically each one of these base ones is one of the things at the bottom of the grammar and then we get to something like this which seems ridiculous priced an estimated utility consumption site trait but you can see that as it extends system design trait with system price with production estimate we can we can place it in a very sensible place in our hierarchy of productions and say well this is the thing that we're we get right before we calculate savings this is the input to the savings operation which you know presumably might be really expensive or something we might have to do that asynchronously so if we would name other products of this grammar and we we could say price an estimated system would be one of those and you could just you could label any vertex here so the quote then ends up being the thing that extends the price to an estimated thing utility consumption say with the savings and now we get a big value that is the product set of all the others same here with installation extending the utility consumption site the system design and the solar production and crucially i included this to show that we can leave things behind we can forget about the savings at this point and still you know we don't require it um because we might not want to impose that restriction on ourselves in every situation if it's installed even if we don't want to generate or price a quote anymore we can still do this so those were the models that are accumulating the traits but services need to provide them and move the the state machine through the expensive transitions so looking at production estimation service this would be a function from system designs to production estimates and crucially this element this service needs to know nothing about anything it doesn't care about including utility consumption sites and this is a very simple operation just for the demo purposes but essentially you know imagine this was really hard really a complex transition similarly we have a pricing service where you can provide a function into a future of system price from system design and similarly when we're doing that savings calculation we can include all of those traits that we defined as necessary for the quote and or priced estimated utility consumption site to savings so looking at why this pays off when we're doing the application of a system design say someone has outside of the api completed the design we can compose the all of these transitions into a four comprehension or whatever so it's a reactive stream flow anything that successfully carries the state machine through the transition and here you can see the utility consumption sites getting found the pricing service is making a call the production estimation service is making a call and the results are being composed those two calls could complete in any order and then finally after they have the savings calculation can be made and we compose the savings onto the price and estimated site into the quote and that corresponds to movement through these this kind of a state machine and we can say that our code has a direct you know more or less direct denotation with functions between actual domains how does this kind of thing look well we use siren which i won't explain completely but basically it's a json structure that can describe its class and it can describe more than one class so once you get back your full quote it can show you that it's a member of all of those domain sets and it defines all the properties that any one of them requires so similarly sorry when we post a solar production site to make an installation we can do a very simple comprehension and say all we really have to do here is is compose find the quote and compose it with the installation and we know that we have a fully valid installation so this is the moment where we actually had knew all the all the elements of this unit oops and here's the result um this siren entity is saying i'm a member of the domain installation i'm a member of the system design you know i'm also a utility consumption site and a solar production site and i have all the values associated with it so mission accomplished we've done a loose semantics of our code and we know what the different parts of it mean and these denotations provide an anchor point between code and meaning a function from code to semantic domains basically the program and its programmers can communicate completely incorrectly with humans about representations that are about how its representations are composed and we have a pretty good positive proof in quotation marks um that it is correct for the whole domain not just for tests on specific inputs and there's a whole lot of uh caveats here how do we persist this don't we need to you know model events think about streaming um i could explain a lot more about how we serialize with siren and how we deal with containment and it shouldn't be mistaken for a real mathematical proof of rigorous denotational semantics for one thing we haven't explained the semantics of scala or concurrent programs but i think it offers a higher degree of certainty than we might otherwise have about the program's meaning and a way to communicate about it in real language so that's it the slides will be online somewhere and here's some links that i found useful and you can follow me on twitter if you're interested i swear i didn't make up my username just for this talk anyway thanks very much and now let me introduce my colleague suresh hi everyone i'm suresh prakash senior software engineer at san gabriel um anybody has experience with the play framework can you rise hands oh so so many of them okay so when i started with the sensitivity uh like i've seen we have a api platform project it's it's a big api it's it's it will turn out to be a big api platform project so i saw that like uh there is a potentially list can turn out to be a complex large applications so i explored the options of creating uh services uh a kind of sub modules inside the uh the big player iapa platform then i find a very useful feature in play framework it's a creating sub-modules so i have created a small demo so the contents are going to be i will explain the background and the folder structure and there are a couple of demos so why the sub modules so as i said before we want to introduce reusable modular separately deployable services and the sub model supports modularity you can separate the modules by functionality and then you can also create a play module or as well as the non-play module so if you enable the play scalar plugin or if you're using java play java plug-in it will become eventually it will become a play module otherwise it's a non-play module so the scenario for non-play module is you may have a utility methods or you want to separate out the your business logic which is not at all related to your play application basically so we are using play as a rest api layer so you can move all those logic into a non-play module and package them deploy them use reuse them as a library so the next point is reusability so multiple play applications it offers multiple play applications inside this inside the the one big root play applications so this is the typical play app folder structure as you know so the root app has controller model view mvc we have account folder if you see there is a protocol modules so play support sub modules you need to create your modules under the models folder so going so this is the demo i've told you before so the root application have a app folder and he says the comments the commons is a non-play module so i didn't enable the place caller plugin and if you look at my build spd which is in the root root folder so i have two uh play modules module a and model b i have a commons so which is not a play module so i have enabled plug-in scala plug-in place color plug-in for model and model b and my root project depends on model a and model b sometimes you don't need to you don't want your root project to depend on modules or modules so i have a two modules and the major part of the player framework is the conf con file your con file and your routes file the routes file has your api endpoints so these are all the endpoints i have in the root app and i i have configured two other routes file so playable route the endpoints which are not available in this file to the corresponding models model a routes or model b routes and if you look at the con file of the model a so i have included the the root application con file sometimes you may need the configurations from the root app so sub models may need to access the configuration from your router this is how you do that and i have a router so in case if i package and deploy the model yes separately this router tells the play to where to route the end points this is the model layer routes so i'm switching to the browser i'm going to start the play app it's already started so it's i got the response i hit the endpoint slash index i'm using postman as a respite api client so i'm the root application so i'm going to enter endpoint which is in the model a sorry so i'm the model a i got the response i'm the modular application so this is going to the model a routes if i package and deploy the modular separately this should still work i'm going to show that so before showing that so model b it says model b application so if you see i have an end point to read the configuration from the application configuration file so there is only one configuration in the application.conf of the root play app and i have uh one configuration in the model year application.con file i'm also reading the configuration from the root play up so we have two key value parts here so i am going to now package the application so here here you can see spd packages module separately so we have a model a model b and the root tab so let's see the model a so the play application from model a is up and running now i stopped the play application from the root application so going back to the browser you can now see so model a is packaged and it's running separately so you can only see the configuration from the model a so it's not reading the configuration from the root tab let's check the other endpoints yep and going back to the ide i have a one more end point which is which is actually is using the implementation from the commons model which is a non-play module so i have a method implementation it's just a util method it is used by it's used by the controller in the model a yeah you can see that here so now model a is up and running separately let's go back to the browser so i'm going i'm going to hit the endpoint which which access the comments module it's a non-play module so you can see i'm splitting the url into protocol domain and top level domain so going back to the slides we have seen some demos how the folder structures are organized and what are the benefits of sub modules so realistically we may uh package and deploy the services separately or reuse them as a library non-play module so there are lots of benefits the one more thing we can you can have a clear separation between your modules so i have a test in the model a so if i run run this test with the coverage it's going to show me the coverage only for model a classes so creating sub models in play framework really helps us to maintain modularity overall and the reusability thank you guys any questions okay thanks you