Scale By The Bay 2019: Niole Nelson, Integrating React with Scala...

Scale By The Bay 2019: Niole Nelson, Integrating React with Scala...

Recording: Scale By The Bay 2019: Niole Nelson, Integrating React with Scala...

[Music] so my name is niala Nelson I'm a front-end developer surprise surprise what am I doing here at Domino data lab I guess you could consider me full stack but I'm definitely mostly from in Domino data lab is a start-up that enables data scientists to do their work without having to deal with building infrastructure for things like model deployment or model versioning and it enables them to collaborate better through we also the structures built on top of kubernetes so if your engine that we are hiring and so hit me up on LinkedIn you can find my details sket IO so right why are we talking about injecting react into Scala play well a couple years ago Domino data lab hired me to help modernize their front end in a general sense and what I found was a kind of a smattering of technologies of course Scala play because the web apps built in Scala play which uses a tempting engine in order to render its UI so you can actually stick a client-side rendered like JavaScript tools in the in the Scala play front-end so that allowed use of things like Twitter bootstrap and backbone.js if I'm sure well I don't know has anyone heard of bones yes I mean kind of it's been its around yeah okay jQuery just like a fun patchwork of stuff you know quilt of aging front-end technologies you know for example backbones pretty much dead now I think even the creator said please don't use this anyway so okay great we have a front end but there were you know there are a bunch of problems with that so of course you know Domino data lab like we have this application it works people can get value out of it it can use this model out of deployment automation system that we've created this collaboration system that we've created but in general this mattering of front-end technologies created a really poor development experience for back-end in front of developers alike you know this is at the time you know when I was a couple years ago when we're kind of it was kind of a disaster the most of our developers were back in developers you know what back in developer wants to write jQuery I mean that's not what you were hired for so why are we making you do that and of course you know Fred developers it's kind of like oh it was kind of disorganized at the time initially I was hired just to help modernize their stack but along the way along the way to like achieving that I we actually uncovered some insidious problems in traveling Domino from a product perspective as well as as an engineering perspective namely poor developing you've got four developer experience and the slowing ability to implant changes according to product vision due to poor technology choices which it's kind of like accumulated over time so um let's just break it down a little bit the poor development experience let's break it down into the cut into a couple categories of disorganized untestable and just generally low-quality situation the amount mission of the jQuery and JavaScript usage and the occasional backbone.js like many web applications kind of like scattered around and then using a Twitter bootstrap for the UI along with some CSS contributed to for developing experimental II there was no core pattern to guide developers when they were adding or updating a feature this was as much a product of the tools and use as well as how they were being used on opinionated libraries like jQuery and JavaScript or the prevailing front-end technologies in some parts of the front end rather than being used as like spitting clue they were more like the supporting structures if you will as a matter of fact as a tire talk probably would never have happened if the entire front end was written in backbone oddly enough there are some problems backbone which we won't really get into but I mean there at least would have been some structure you know some shape to the front-end had that been the case when you I don't know a few friends this but when he used like you know like low-level there no low-level but you know low-level front technologies like jQuery and JavaScript as the supporting structure you're really putting yourself at the mercy of a of your developers creativity and which is you know don't get me wrong it's great to express yourself but you know it can be it's not exactly always the best thing I'm sorry don't crush anyone's developer dreams so it's a it's a problem because you really have little to no guarantees as pertains to creating an understandable or organized or testable code base and that's what's exactly that's exactly what happened the lack of organization led to an untestable code base which was hard to understand Hart on board too hard to change easy to break possible only to test with end-to-end integration testing using browser automation frameworks like selenium or phantom Jas for example which are notoriously brittle ways of it's a notoriously brittle way of testing because it's nearly impossible to get a large production level web application into like a dependable state when you're going through your test cases then also when you don't have good separation of concerns in this case of view rendering logic versus the business logic of like sticking stuff in a database or computing information which will be consumed by the front-end but doesn't that stuff doesn't really need to exist alongside each other you know so over time you kind of end up with this like a massive monolith it's like a massive codebase which takes a long time to build like in a CI or even locally if you're gonna have this like a local dev setup where you pull down like everything it takes a lot longer when you have to build the backend in order to make make it so the build at the front-end can also be compiled on the other hand slow product evolution is like a big issue that we ran into if developers are not properly enabled to do their work naturally the product also suffers when you have a UI with no organization just like a smattering of Java scripts everywhere it's nearly impossible to refactor that to a shared component library and a shared component library is the basis of evolving your applications use your experience in like a sane and timely manner with minimal breakage and minimal bug fixing dependence also on global context for styling and you know in the case of CSS makes product related changes generally difficult dependence on Twitter bootstrap if Twitter's in the house I guess they're one of our sponsors or something no shade to Twitter bootstrap you know their bootstrap library but um if you use like Twitter bootstrap for all of your styling you're pretty much married to Twitter bootstrap and if you haven't been like working on making some kind of share like reusable way to share styles or something like that in order to like change the food share components to like look like what you might pride to look like then you're pretty much screwed in the long term and not to mention if you override you know if you're basing your entire styling on like a injected shares they're pretty component library like this and it's not like you're not being careful about how you're styling it you're pretty much just making a throwaway style changes which is a big deal if you're trying to like level up your organization from being like okay great we got the MVP and you want to move on to become like have like a real nice production application where and iterate in order to improve what you have in order to support customer use cases and grow as a company ultimately this stack is not developer friendly no one wants to work in an application that where it's hard to provide value we all want to be generally productive human beings do good things make something that we're proud of we want to be able to hire front-end developers who care about the front-end concerns and know how to develop the front end and in the domino case we wanted to move away from burdening our back-end developers with work that they didn't want to do and work and we're not hired to do also relying on one developer to do full stack development for all features when you could paralyze the work makes no sense especially when you're trying to grow an organization you know full stack development is hard and we want to hire front-end devs to speed up the evolution of our product so that was our general goal which lettuce you know so those are all the questions and problems that we were that we the problem that we had the questions we asked yourselves like how do we kind of incorrectly move out of the sticky situation without throwing away everything so we thought a bit about that and we kind of decided to cut a cup with some sort of incremental update approach to move to the new paradigm which should be a front-end back-end split the concrete example here incremental e separating the system smattering of JavaScript technologies from our scholar play back-end well from what would be our Scala play web application which would be more like an API server and then which would serve the front end and also may be used to have some nice api's for for CLI used or something like that so we kind of like we thought about it for a bit and given that you're able to execute JavaScript in Scala play in the templates we we kind of rolled along and came up with a pattern for injecting react and just call a play client-side rendered reacted to Scala play so we're going to talk about how this concrete example solves solved our problems at Domino but really you can probably extrapolate this idea to any monolithic web application that you don't want to completely dismantle and rebuild the UI from scratch boat suffers from the same kinds of problems that we just talked about like unto stability disorganization general confusion and just pain a world of pain okay so so in our example we've chosen two incremental e move to use react as our front-end framework to solve our problems with the three points I just mentioned and then also to help us move more quickly on certain axes of product development like trying out new user experiences quickly and quickly enacting application Y design changes so in our case we have chosen to use react because it's general it's it's the opinion that it brings is one of general modularity which is exactly what we're looking for and so it'll help us do create some kind of sane way to do incremental changes but also build up like a shared library peace pipe peace like from the ground up but none of it will be wasted work because we will have created something that's shareable and that we can take with us and evolve without spending too much time on it ultimately the reason why it'll work from a technological perspective is because react so when you see like a Docs or examples of react with the JSX syntax well that's just all syntactic sugar and we have all these technologies in a front-end lamps you know in the npm ecosystem all these packages that are used to that can be that serve the purpose of transpiling this GSX syntax down and to just plain old JavaScript which we can stick right into the Dom because we're quite lucky that browsers can it can a read and execute HTML and then know how to download and execute JavaScript so that's our secret weapon in this case and that's what will make it so that we can use react that's called a template also it's a piggy national giver developers some structure and it'll attract front developers for a company that's a win it has a large ecosystem with lots of conventions surrounding it for testing for developing UI components in isolation so you know meaning you don't have to deploy your your application in order to develop these individual components you we can we can use this this wonderful tool called storybooks if you've heard of it so like say I'm making a new modal I don't have to we have this little toy server that we can use to develop this modal we don't have to serve up our application and go to the right page in our web app in order to look at it and iterate on it and also it reactors efficiently typesafe and with the introduction of typescript in the past couple years it's become a lot more mature and it's totally production-ready and it's honestly like takes a huge burden off of you psychologically that you don't necessarily have to just like click through every you know every time you touch a piece of JavaScript you don't have to like click through every single workflow in order to make sure you're not gonna get fired tomorrow when you merge this honest I mean honestly the other day I was um for fun I was just creating um a birthday card like website for my friend for her birthday and I was like oh this will be just so fun I'll just like inlay the JavaScript and I'll just inline the CSS and I'll be all fine and then um you know Joseph just lies to you it like it like it's like okay I'm gonna run now and then it just like you're like what happened it didn't nothing's working so use typescript all right so react it's opinionated think so react you can kind of when we think of react I'm you I don't know I don't know any of you guys I don't know what your experience is with react so we're just gonna assume that you've seen pictures of react components occasionally in like hacker news that's I'm gonna assume for you guys so you can make a react component the great thing about react components though is that so they're modular you can also render more react components inside of each react component so ultimately you actually end up with like a tree of react component which is kind of lovely in itself one of the great things about it is this idea of unidirectional data flow so assume this um abstract representation of this tree of react components let's just pretend it's like a drop-down you know the root node is the entire drop-down but then some of the nodes below it you know maybe this one on the left with no children it's like a button for opening the drop-down and then like on the right side like that's just the drop-down part with all that you know the list elements so kind of nice to think about so that was just to help like help you center yourself in this tree diagram so anyway the great thing about react design its unidirectional as tiful is unidirectional so it that makes it a lot easier to debug when you have a problem especially if you're not using typescript but you're gonna use typescript so but you know back when we just had JSX this was like yeah this was like life-changing anyway so uh it's because the unidirectional data flow exists it's easier to conceptualize what should happen given a given your data looks a certain way so say your state that comes in at the top of the tree looks a certain way you can kind of just extrapolate out what everything else should look like because nothing's like sending events to random places and which may or mean which may be caught somewhere else but you don't know because it's just kind of one of those concepts where the event goes out into the ether and uh gets hot I don't know and you you uh yeah so it's not not easy to follow there's no like stack trace to follow that kind of look that kind of like a pattern of eventing so this automatically makes it better than backbone.js where you can have like bi-directional event emitting infinite loop nightmares where like you know in impact when you just you can emit events they can get caught and then that event can get caught by something else which emits an event and then maybe that gets caught by the place that emitted the original event and then you're just going like you know infinite loop nightmare so you won't get that with a react if you don't add anything crazy of course you know just by itself that's way it is better than jQuery no opinion about which has no opinion about dataflow and it also provides an elegant grotto both structures for developer developers to model their applications with so in general the recursive structure of a react component tree is quite nice to look at and think about in my opinion each sub tree in the react component tree represents a chunk of the view in your application and the entire tree is a webpage but as you drill down to the tree you know one sub tree is a drop down now there's a button and now there is some kind of like sub view lots of fun and you can even have because we have the tools to transpile react down and to just plain old JavaScript you can have actually several independent react component trees on a single HTML page so that's something we're going to take advantage of in our injecting react into Scala play idea and that which will you know help us evolve our front end away from our back end and eventually into a separate server so if you keep your data flow as pure as possible you know if they're you know where you know you keep all of your i/o and all that stuff at the root of your tree then and you keep everything you know basically the idea here is that your Dom whatever is rendered in your in the Dom is a direct representation of the Gerdes the data that you supplied so it's just inputs to outputs you life should be simple and it can be and it is with react if you don't do anything insane like adding random event evading an event emitting for but I don't know why you would do that because that'd be weird so if you keep your dataflow is pure as possible if there's a runtime error like like we've modelled here despite the large number of react component nodes there's actually quite a short path way to investigate in order to understand the origin of your bug it's also performance the just the idea is that every operation you were to do were to actually touch the Dom that's actually quite a computationally intensive operation for your browser but if in the case of react it uses a thing called virtual Dom which I'm sure you've heard about it was it's the hotness I don't know it's it's popular and thank you you know we hear there are some words you're over over and over again I guess virtual Dom is one of them and the idea is that if you just keep the representation of your Dom in memory like as a string or something it doesn't yeah that's actually a lot and then only apply the the actual changes to the Dom that's going to be way more performant than performing you know performing unnecessary updates to your Dom all the time and then just here's just a in case anyone hasn't seen typescript yet it's this is evidence that it's real it's actually here and you can finally get like insight into whether your data that you are you are providing to your react component in the case of this item is actually of the shape that it should be so we for a long time you know we would get if I was just linting and linting is is great it'll but it'll let you um let you know if you've got a variable that isn't defined anywhere but you're attempting to use that's pretty important but eventually eventually that's I mean that's really not enough but that does keep out a certain class of bugs but ultimately it's a it would be so much better if like we knew for sure that the shape of our data was correct when we up supply it so uh I know some of you are probably thinking like why aren't we talking about Scala j/s react come on and I yeah I totally get it or GH CJ s that would have be lots of fun to use in a front-end web application I agree but in all fairness you can probably come up with approach that works for you with any framework that will eventually compile to JavaScript which would allow you and that would allow you to separate your friend and back-end just fine in this particular situation so yeah go for it depending on what your overall goals are Scala GS g HC GS that they may get you there but in my case I know there is an ecosystem around react J s which enables a certain profile of typical JavaScript developer and I know that certain tools exist that will be so helpful gos to any you know develop quickly address product requirements quickly so the decision to use react here yeah I like I like react a lot but it's as much a strategic decision a business decision as anything else so all right so let's get into the implementation for how how we are actually going to do this how are we actually going to separate our back and in front end how are we actually going to enable these faster build times so let's get into it so the magic all boils down to two things reverse proxying and taking advantage of the fact that when you're writing a web application no matter what you will ultimately be executing javascript in HTML in a browser because that's how it works in the in our case the role of the reverse proxy is to allow us to sprit the front put the front end into its own server which can have like Harvey loading in dev mode super important and then in prod depending on your setup and one of your limitations are you could or probably also depending on your relationship with your platform team you could bundle the J's a set with the main application or have it served stat as a static asset on your your separate note server which will be proxy to in order to maintain the current user experience for your end users and the setup would look something like this you've got your a chromium your new chromium style internet edge explorer web browser over here user opens up their their cutting-edge Microsoft new edge browser and they decide to look at your applications homepage they hit slash home and a reverse proxy intercepts this request and it sends it in this case to our Scala Play application which knows how which which responds with them like that that's the template which represents your home page and the home page happens to have some script tags in it some of which reference our vanilla JavaScript and CSS assets which are which are our like shared react component library that we are creating so when those script tags get executed the browser will then send some get requests to slash assets blah blah blah that'll give routed through nginx again but in this time it'll be sent to your web pack dev server where you're doing JavaScript and which is serving your bundle your jeaious bundle and your CSS bundle or whatever you want really I mean it's just a little server so in dev you know hi reloading has source maps for debugging in prod assets are compressed there it's a you could really do anything with this tiny little JavaScript server in any case the user experience is preserved but we're inching closer to our goals with front and back and separation all right so what are we gonna do how are you can get the JavaScript into the Scala play template we're going to create a like a helper utility which is going to have act be able to reference our shared component library and just make it available so it can be mounted in HTML and this is so like in practice the use of the injection helper is gonna look a bit like this you've got your reference to your bundle which is you know we're going to be proxy to the correct server in order to get the correct assets and our rack our component libraries going to expose this global variable called component library on the window and it's going to have a little helper function on it which takes some important specifications like what's the name of my component and what are the props and what node should my react component be mounted to so you can have like you can specify where your react component goes where it gets used and then this is the internals it's really quite simple we basically we go and there's we've got some sort of import called all of our all components and we go and get the component based on what the name is that was supplied and then you know if it exists we just use a react Dom to mount it in the place that we expected and I got some code all written up we're probably not going to go through it but I'll post my slides on skid I oh so if you want to like go follow the links that'll be you you can do that and there's a if i and i have a little demo running but I'm like totally mind blown by this um two screen setup so I don't know I probably can't so here's the evidence that it actually works here's a tiny little application we've got our nginx configuration running it's um and we're look we're looking actually at a page served by a scallop play web application with some things in play as you can tell by that string up there that parts in play but then these tabs they are rendered by a separate server which which the reverse proxy proxies do so that we can access those assets all right that is that is that that's how you do it thank you for listening thanks Noel any questions let me ask you sorry I you had everyday or anything I said you know why nots College and yes sir do you know that's very bright but so my question is why not react 'once colleges oh yeah I know that the thing yeah you should do it I just don't I didn't get a chance to do I didn't get a chance to use it but I've seen the the syntactic sugar looks nice and fun and apparently the concepts are all there so yeah these guys getting better I'm like we're I'm actually in a project that we're making lots of progress to make more even more wrappers around it oh cool so if you want to talk about it afterwards all right yeah sure have you a cool thing any other questions for Neola all right thanks a lot thanks everyone [Music]