scala.bythebay.io: Brennan Saeta, Productive technology at scale with Scala
Recording: scala.bythebay.io: Brennan Saeta, Productive technology at scale with Scala
you so thanks all for joining this talk today this afternoon today I want to talk about Coursera and how we've grown some of our worst blunders and some of the magical discoveries we've encountered over our four and a half year journey so far for those of you who aren't familiar course Sarah's mission is universal access to the world's best education we were founded in 2012 by to Stanford professors and we partner with world-class institutions from almost every continent we help them take their best courses from their best instructors and we put them online so that anyone around the world can take these courses we have over 20 million learners we have nearly 2,000 courses from over 150 different partners we weren't always this big when we started we were actually a very small team we could all sit around a table and we could all yell at each other whenever someone broke the site which happened every once in a while today however we've grown quite significantly or over 200 people and we have over 65 different engineers and along the way as we've grown we've realized that we need to change out some of the technologies that underpin the Coursera learning platform we've evolved as some things have worked well for us and some things really have not and so that's what we're going to talk about today but to understand the motivation for why some technologies have worked in some technologies haven't it's all about scale now there's the typical scale you know dimensions if you will of like queries per second handling a huge amount of load or concurrent users or handling terabytes or petabytes worth of data in your databases but actually while we do have some of these sorts of scale challenges we have scale challenges in other dimensions other dimensions that I actually think are harder to handle than these purely technical ones and the most important one I want to talk about today is the number of types in our ontology now as for what i mean by that i think is actually best illustrated with an example if you're a messaging app and there are great messaging apps out there you actually have very few types in ecology you might have messages you might have feeds you might have users and while there can be a lot of work that you've done that can be done to power these sorts of things there's relatively few types in your ontology relatively few things or categories of things Coursera on the other hand is multiple orders of magnitude larger so if we just look at Coursera is catalog the out of course experience we've got courses instructors university partners we also have course lists and specializations in course domains and we have a huge number of other types surrounding that but then actually most of the products actually in the in course experience you've got video lectures you've gotten video quizzes you've got assessments you've got peer review you've got programming assignments you've got forums there's so many things that are part of the Coursera course platform there's so many types but actually course there is even more of an iceberg there's a huge amount that happens underneath the hood for authoring and versioning of courses in grading and so much more that happens underneath and Coursera I think has a huge number of types in oncology and that's the combination of these three dimensions of scale a more nuanced understanding of scale is what's going to help us trace through what technologies have worked well for Coursera and what happened so we're going to go through and we're going to focus on three key technology areas or persistent systems how we know how we deploy code and how we write ap is so without further ado let's get started so we're talked about persistent systems when Coursera course are was founded we inherited a PHP in course platform it was written by a few grad students and undergrad students at Stanford and they wrote this basically over their winter break and had a whole lot of fun doing it and a lot of caffeine as well now this this PHP in course platform it was written with basically custom everything there was no libraries used there was an own a custom in-house equal sanitization layer own query routing layer it had its own patterns had 0 unit tests and it had zero documentation now when I joined you know it became pretty clear PHP is a fractal of bad design right just a horrible system to do anything in right clearly that's obviously true and and you know we thought we really had to move beyond this and so we looked around and we realized hey you know a lot of us like Python pythons you know pretty good language and and Django is a really powerful web framework it has you know an ORM it has a query routing layer it has powerful plugins for schema management JSON templating like we practically just had to sprinkle a little bit of business logic on django and like voila our app was going to be written we thought that this was you know clearly the solution to all of our woes what were we wrong so we did have some concerns picking Jengo in Python we knew that you know python and django very slow language in comparison to scala or C or otherwise but we thought scalability schmale ability we'd have to worry about that this is horizontally scalable we just throw more machines this is just trivial operations on a cloud provider you just make an API call and grow more Hardware thermal hardware the problem and and I would solve all of our scalability problems while we were true we've never had scalability problems at our CPU or memory layer in our application tier that was not that was not where we encountered the scalability problems we encounter them all in the database layer and I think to understand what was going on we first need to look at some numbers most users spend most of their time on the in-class experience that's the catalog guys just to get to the in-class experience so most of our activity most of our data most of our queries were happening on the in-class platform and we we had these sharded databases we gave each database about 15 gigs of RAM that was sort of the ec2 instance size of the time and they were serving about 300 gigabytes worth of hot or warm data by contrast our catalog are out of course experience we only had 68 sorry we were serving just 20 gigabytes of data but we had to run on our machine of sixty eight gigabytes worth of RAM just to keep it alive and understand why why we had to so massively over provision hardware for such a small data set I think it's best illustrative if we look at an outage so this is sort of one of our worst outages that we had but but helps really understand what was going on we rolled out a new API Wednesday evening relatively low traffic time period we started using it on our site we made sure that it all worked and that was great but what we didn't realize at the time that we radically increase the number of I ops that our database used so afterwards after our outage we rate we what we ended up doing is we ended up launching a big media marketing campaign to say hey leave got tons of new courses come check them out tons of users flooded our site and we just were totally down for about four hours but what happened actually is that this new query was joining about eight different tables together in my sequel because we have so many types in our ontology that my sequel query optimizer even though it had perfectly enough RAM to fit all of the data in memory it ended up spilling intermediary temporary tables to disk and this had end up radically increasing the amount of I ops and when we increased load we were just totally host this is an example of why types in your ontology or the number of tables in your database schema is it particularly difficult dimension to scale what's important to understand from this is that it's not the Django ORM that can never scale or handle huge queries per second or huge amounts of load it's at the jingle or M fails when you have a large number of types in your ontology so what technologies have worked well for us wow you really can't see that sorry about that we use slick today slick allows us to much more clearly express the queries that we need to send our database ensure we're not sending o of n queries to our database ensure we're not joining together up to 12 or 13 different tables which we've easily seen in our Jengo slow query logs slick works very well for us but before we adopted it's like we actually picked up a norm from the play framework and that also works very well for us but to be honest as much as we made fun of PHP as being a fractal of bad design and all this in-house you know custom framework as being impenetrable in horrible we never had a problem with just writing sequel and running it against our databases so that's an important lesson that we've learned not all tools scale equally well across all dimensions of scale now as our product evolved we realized we needed to change a few fundamental assumptions we previously ran courses between three and four times a year but we realized that's totally brain-dead this is the internet you should be able to start a course whenever you want you should be able to go as fast as you want like this is obvious why are we still stuck in you know 14th century terms of classes and so we realized we need to change the way our in course experience persisted its data we could no longer rely on our session based sharding strategy we needed something that could increment alee rashard with zero operational overhead and the only database that we felt comfortable with at the time was Cassandra Netflix had proven that it runs reliably at scale on Amazon ec2 it can handle a huge amount of punishment both in terms of course per second as well as in terms of loss of instances so we picked Cassandra we started building on top of Cassandra naturally Cassandra's worked because Sandra has absolutely delivered on the promise of availability in criminal scalability and relatively low operational overhead we can upgrade the database without taking down time in ways that we couldn't do with sequel but it actually has its drawbacks too we've found that most developers don't interact with or touch Cassandra directly it's much more effective for developers to use simpler higher level of abstraction and not to use the full big table data model that Cassandra exposes we instead use some automated key value abstractions and that actually powers the vast majority of course Sarah's platform today but Cassandra doesn't power our catalog our catalog is not a huge scale problem our catalog is only a few thousand courses a few hundred instructors and a few hundred university partners and an important lesson that we've learned is you don't want to pick the most scalable most awesome tool for every problem don't use the same hammer there are some problems that are really small in certain dimensions of scale our catalog only a few hundred megabytes to just serve in RAM but we have a huge amount of queries per second we use our catalog API for more and more places throughout our site so what we do is we actually load up the entire course catalog into Ram on the JVM heap of our catalog servers and we just serve right from in-memory this works even better than a cash for a number of reasons in addition to getting lower latency and no network hop we have true horizontal scalability we can incrementally add new servers to our catalog tier and serve more and more queries per second concurrently with you know ad infinitum additionally you have much better availability and split-brain scenarios if you just have everything in RAM you have a unified failure model and you don't have to worry about partial failures between your caching tier and your application tier so this has been a really big lesson learned for small scale problems or other problems that have a different point in the scale space so I previously defined three dimensions of the scale space queries per second data set size and types near ontology you don't want to pick the same tools in the same technology to solve different points at different point to solve different problems at different spaces on a scale continuum so with that let's move on to our next topic of area deploying code now as a number of different startups may have started out it's quite common to just do a get pull on your one application machine and that's how you deploy new versions of code that's exactly how we started except where we then had this whole PHP and Python thing so we had to do git pull on two different machines and got to be a bit of a mess so I wrote deployed on SH and what it would do is this a shell script that for so well made sure you're running in a screen session because it's quite often you're going to lose your ssh connection you don't want to have the deploy fail in the middle of or have your connection fail mental deploy and then afterwards it would you know automate association to all the machines and effectively do a git pull but at least it was behind a script and this is a huge improvement because it meant that instead of deploying like once in a blue moon we could deploy maybe a couple times a week and so as a result what we ended up doing is we actually unified our PHP and our Django under the same boxes so we could have one script just consistently if you have a problem just run deployed SH it will restore the world to a uniform environment we had a monolith or we had a stereo lift that we converted to a monolith that was definitely better unfortunately deployed sh really wasn't adopted that much we wanted to be able to push out code much more frequently we were to have all our developers to play their own code and so we built this web console that we call Wayland Wayland allowed you to with just a few clicks in a web browser push out new code roll it back we used immutable infrastructure with blue-green deployments and this is a huge improvement across compared to our previous infrastructure we turn on auto scaling and we found that developers would gladly deploy their own code up to a point and we went from deploying a few times a week to deploying a few times a day but we had a still a number of problems with this in particular it was a monolith when you wanted to deploy just a small JavaScript change you had to deploy all the Python all the PHP and all the other JavaScript same wise if you say likewise if you just wanted to deploy a small bug tweak or an improvement in the backend and because you're deploying so much code and end up being a slower more cumbersome thing and developers started stepping on each other's toes and so we launched our second generation deployment infrastructure built around scala microservices or rather we just call them services and there are a number of key choices that we made as we made this leap to scala and to micro services and we took actually a different tack in a number of different areas in particular when you micra service this is quite common that you allow a polyglot approach you can use whatever language you want for your service I'll use whatever language I want from my service and we'll all just not talk to each other and we we actually thought that was the wrong approach for Coursera and that's because Coursera is evolving so rapidly we have just barely begun to understand how to teach effectively online we're constantly reprioritizing andrey shifting as new experiments give us new directions to go in that we needed to have the flexibility to move code between one service and another we need to have the flexibility to move one engineer to another service and to move services between teams and so every service at Coursera is Scala it all looks the same that all functions the same it has the same built in integrated monitoring so that every service can be monitored in roughly exactly the same way we've found that this has actually worked incredibly well for us Scala is a general-purpose language it is powerful enough it is scalable is fast that we have no need to have additional languages and cloak ated into our stack and this has been an important simplification for us in our second generation the other big improvement that I want to talk about is that we realize that you really need to keep a very low latency between commit and having your code running in production developers running their code they get review they go back and forth they commit they should then immediately be testing it in production and making sure that it runs reliably and so our target was actually six minutes from code commit to having it running in production we've mostly achieved on that and this has been a big boon in developer productivity and maybe most importantly developer happiness now today we ended up building all of this with custom in-house tooling but today technology is advanced and these are sort of the tools and technologies that I would take advantage of if I were starting over today kuber natives and docker definitely very powerful there's already a lot of existing tooling around it to support these immutable infrastructure Bluegreen deployments and auto scaling definitely check them out if you're not already but what I really want to talk about is that these same principles apply not just to back-end services but they apply to the front-end code as well we started we used to deploy our JavaScript you know with the same back-end services but we have actually divorced it and so we have our own what we call rapidash that allows us to deploy code in JavaScript just the same way immutable artifacts immutable versions we've actually integrated it with our unit testing framework and with our monitoring and this allows our front-end developers to move with much more confidence with and much more quickly there's a built-in control Z so if you break the site it's less than 10 seconds to roll it back but similarly in the same learnings from microm monoliths to micro services we've done the same thing with our JavaScript code we've had numerous single page apps for our catalog we have another single page app for the end of course experience another single page app for the authoring environment and we have other single page apps that are internal and so we have rapid dos version 2 point 0 of rapid dash that is used allows us to break up our front-end monolith and deploy it with immutable builds independently and allow teams to move as fast as they can or as slow as they need to to ensure reliability and consistency so taking a step back these are sort of our our key principles and learning from dealing with a really broad wide platform large number of Engineers iterating independently laundry locations microservices let teams like our assessments team move as slow as they need to users get very very upset if they fill out a whole bunch of questions and an assessment and try and submit it and you know oops doesn't work like if you're uploading a picture or whatnot to a you know a post you know if that doesn't work you may say oh I'll go I'll just come back five minutes later you just worked for three hours on an assessment if it doesn't submit and it's ten minutes till the deadline you don't just not care you really really are upset so we are assessments team was very slowly and carefully and they can do that enroll things out carefully because they're totally independent from some of our growth teams that are iterating very rapidly and our other pedagogical teams this has been very important so the last topic I want to talk about is how we handle api's this is maybe one of the most interesting there is i'm most excited about this is actually where we really touch scala so our PHP environment was web one point oh it was you know sing server-side rendered pages from PHP you know old school it worked but it's boring so as we started growing you know we had fun in DES and like no no no no web web bumper no no no let's do single page apps guys javascript is the way of the future and so like all right fine whatever what do you want to do this like just give us JSON API is Jay Sanjay Sanjay Sonic all right fine so we started building JSON API is in Django and this worked for starters very effectively there's built-in support for JSON in Python and and that was great but as we started building more and more complex api's as we as our product became more and more feature full we started pulling our hair out we'd have dictionaries of lists of dictionaries of lists of dictionaries of dictionaries of dictionaries and we'd have no clue where we were in the type hierarchy is we were trying to assemble this massive JSON response we realized we absolutely needed static typing just to have developers understand what was going on in the code base so we played around with go we played around with Java we settled on Scala as the language of choice for our static typed microservices and we realized that play supported by type safe but but now light Bend was was going to be the the micro service of the core framework that was going to power us play was you know powerful its general purpose it's fast it's reactive although we didn't call it reactive at the time has non-blocking i/o we thought that play was going to solve all of our problems and we were half right the way we started adopting play and scala we had our existing stack over here on PHP and Python and we decided you know let's test this out carefully so we built our a separate system basically totally independent in play and Scala this is our event tracking system there are a huge amount of pedagogical insights to be gleaned from realizing how frequently people rewind videos or where where they get stuck or what questions they get wrong so our venting system is that framework that we allow that gives lets us show our instructors pedagogical insights to help them improve their courses continually not worked very well so we're like great let's build more stuff in Scala so we added be rude are we rewrote our authentication system in the Scala and we wrote our notification service that handled all emails and mobile push and a bunch of other different things as part of our notification service those went great so we're like let's write before you can write more things though we need a way to expose the api's we wrote in Scala directly to our directly to our web and mobile clients and so in order to do that we wrote a routing layer that we call edge that allows us to take requests that come in from our clients our web clients are mobile clients and our third party oo oo to based api's and route them to the right service and this basically unleashed a whole number of new microservices sort of services written in scala we have our assessment service we have our catalog service we have our course content service and our authoring service and as we spilt more and more api's we realize the limitations of play and the limitations are perhaps best illustrated by looking at the routes file so there are a few things that you can see by looking at their routes file this is almost exactly taken with maybe some slight modifications from our source tree and the first thing is that it's actually a little cumbersome to write a lot of different REST API so it's totally possible you can use all the HTTP verbs you can have whatever HTTP query mapping you want but you end up having a lot of repetition for example we have you know utilities repeated a bunch of times there / announcement / v1 a bunch of times so it's a little cumbersome and actually this is abbreviated for our wiki's we've got 15 entries in our routes file we've got over 10 for the utilities a half dozen for mobile down here at the bottom and dozens for announcements additionally if you look carefully you'll notice that we're starting to get a little bit of bike shedding the first two lines represent you know the first two announcements API wanted to get all of the announcements and one is to get just the visible ones and we actually had a lot of contention should the visible be a path parameter should that be a query parameter should that be a header you know it was that definitely unclear additionally we have a huge broad number of API is a huge number of component our platform if we have to write them right the api's for them three times once for iOS once for Android and once for web you know that's basically a total non-starter and you see us starting to write specific mobile api's and we were afraid that of what that might lead to so after a number of iterations their number of false starts and after a number of failures we wrote nap time nap time is the product of our learning of how to build AP is as effectively as possible on top of the play framework in Scala nap time is of course open source you're welcome to play around with it and use it yourself it's Scotland native and that's what I'm going to talk about the rest of the presentation so this is exactly what a sample naptime reason if you want to build an API in nap time this is literally all the code you need to add a new API so walking through it you know you define your your resource we call them resources instead of controllers but similar principles apply you inherit from a supertype that brings in a bunch of helper methods that makes it a lot easier for you to work with stuff you have two lines of boilerplate that you need to write you have to define the resource name you have to define the version everything is versioned after that you just write methods just like you would in play you don't have to write any routing code you just have to bind it with juice and voila you've made a nap time resource here we've defined to nap time methods or actions similar to play actions one of them is a simple getter naptime yet another one is what we call a finder there is a strict ontology for what ap is our how they can be represented what you can do with them that I'm not going to go into today what I want to go into however is some of the learnings that we've had as we built nap time so this will not be a guide for how to use nap time to write your next web service this will be a guide for as you build your own libraries and as you use libraries here are the things that work well for us so the first thing that I want to talk about you actually use a little bit specific we're very specific in our types now if you have like an end point like / AP I / courses you might think that you should actually just have one type as the type parameter you know the thing you're returning the course type we realize that's actually totally wrong in addition to making it very difficult to work with persistent systems where you might want to only store the body as a key value type thing we found that it also is very frustrating when you're trying to do writing or mutating methods where you need to supply a body but not the ID type the server may be generating the ID type so when you create a new course we generate a uuid that corresponds to that course we want to get just the course name the course slug the course description we we don't want clients to assign the ID and by separating out the ID from the body type that radically simplifies the types that you end up working with so we model everything as a collection of T value pairs at the type level additionally because we have the separation in types because we have very specific types we're able to catch a number of errors at compile time so if you as a developer accidentally wrote int for the ID type but you've configured string at the top we actually don't compile your code we throw an error there is no way to have inconsistencies in your ap is at the type level this has been very important it helps developers catch issues before they hit production now how do we do this because there's no known natural Scala way of doing this this bind resource even though you might not realize it at first it's actually a macro underneath the hood and that's going to go through it and when it's generating the routing code for your resource at compile time it performs a number of checks and the errors show up right in the right locations where you have the inconsistencies we put a lot of work into this and I think that it results in developers that are very happy with the system so one important lesson that we learned is that we needed to make sure that developers can use nap time without realizing that there's macros or quazy quotes or anything going on or eat the head and these api's reflect that put a lot of effort into that additionally we use a number of other powerful scholar language features we ensure that you have consistency between the types that you return and the types that you've promised in other parts of it so everything is internally consistent you cannot return an instructor in the courses resource you can only return courses in the courses resource and they all have to match identically and this consistency really enables client developers to know what they're going to get before they get it along the theme of type safety catching things at compile time it was very important we realized to allow abstract data types or your arbitrary a user-specified types in as many places as possible in the framework and I think we've done that to a great extent the ID type can be an arbitrary case class it can be a courier class which is course sarah's schema definition language with scala native bindings IDE integration with IntelliJ SBT and play integration but also in terms of quarry parameters you can buy an arbitrary query parameter types anywhere inside nap time resources and this was very important to let developers operate at a higher level of abstraction to catch more issues at compile time and help them work more productively the final thing that we learned is because nap time is so opinionated and so strict we found something that we didn't expect client developers benefited from this strictness this compiler verified consistency very much as well what we found is that client developers would come to our back-end engineers they would say hey I need a finder that takes this query parameter and is on this resource and the that very short brief description let the client developer describe exactly what they want into the back end developer the back end developer knew exactly what they need to do and the client developer knew exactly what they were going to get so having a strong opinionated framework absolutely actually helped improve communication so something we didn't expect so naptime uses a number of advanced language features in order to ensure all of this consistency we use macros we use quazy quotes we use type classes and implicit if path-dependent types in there actually but what's very important we put a lot of effort into is making sure that it's very easy to adopt if you look back at the code you don't see any of those advanced language features right up into your face you don't see very many operator overloading what you do see is your right your method you have your query parameters you do nap dot and you can easily in your IDE explore what are the available options to you with fill ins etc and so we've worked very hard to make sure that developers can very quickly pick up on nap time ap is and as a result we've seen very significant adoption at Coursera we introduced the framework in late 2013 and we iterated on it a number of times but once we got to you know the middle of 2014 developers just absolutely started banging down our door to ask for it we never prevented developers from using stock play in every single place every naptime service is actually a play service underneath the hood and you can mix and match entirely we've just seen a huge take off in the use of nap time at Coursera because developers find that it helps them work better this is because of all the hard work you put in but as a result of this we've realized that nap time has increased our scalability in terms of the number of types in oncology we can now handle the rate of growth of naptime API is because it's so much more optimized is so much faster for developers it catches so many more areas at compile time it helps them move faster that we've been able to radically change the shape of our growth curve in terms of the number of API is that were able to provide an author over time so now time is a very specific course our library it's open source you can absolutely use it we've designed it to be general purpose but I want to take one further step back and really think what have we learned at Coursera as a result of nap time Scala sometimes gets the rep that it's too hard to learn that there's too many advanced features that it's impossible to use and even more importantly that there are some libraries that use these advanced language features that are absolutely impenetrable we've found that that's actually not true if you carefully invest in providing a useful API that's easy to learn that may or may not have all that great documentation we've actually done a pretty bad job at documenting nap time for our users at Coursera but if you provide the right API you can make it very easy to learn despite the fact that you're using really advanced language features underneath the hood we found that type safety catching bugs at compile time things that be caught by the type system and things that can't by using macros is actually an incredibly useful tool to help developers move faster and finally this is ticking even maybe one further step back the right abstractions maybe less general purpose or powerful this actually works across all three of my stories the jenga ORM you can more easily and more efficiently represent more sophisticated queries than in Cassandra or even necessarily you know slick gets pretty close but with the Django orem you can it's a lot more powerful some more general purpose we found that that was the wrong tool deployed on SH that was SS a Qing into machines you could do all sorts of stuff more general more powerful we realize that's not the right tool a custom web console was actually the right tool to deploy code taking away that power making things safer was absolutely the right thing the same thing with play play is general purpose it's very powerful and we realized that we need something that is opinionated and specific you cannot build an entire website with just napped on UPI's our authentication still just stock play but there's so much that you can do with the right abstractions that can make you far more productive and so that's really some of the important things that we've learned general-purpose things may work well for certain levels of scale opinionated things or less powerful things work at different dimensions of scale now nap time has worked incredibly well for our server side components but actually there's more that we can do and so today we're using graph ql to extend type safety from our server components from our nap time ap is all the way to our clients graph ql is a type strongly typed statically typed specification and what we've been able to do actually because Facebook has discovered the same principles that we have discovered while building nap time we've been able to define a bijection back in developers at Coursera will continue to write nap time api's and client developers will use graph ql to access those nap time EP is we have a translation layer a by jection that allows us to translate incoming graft you all queries to nap time and nap time back in a graph kill on the way out and in that way we can take advantage of our optimized Scala AP is and expose them both as rest over HTTP as well as at the same time / graphic you'll ensuring a smooth incremental productive trans transition to this new technology my time is almost up but if any of these challenges sound interesting or exciting to you Coursera is always looking for great people to join our team to advance pedagogy to help us dig deeper in building the right tools for teaching at scale online with that thank you very much for listening to me any questions I think we've got a mic coming over great questions so let's see one thing I forgot to mention i'm sorry to repeat the question the question is what tool do we use for api documentation number one and then number two what tool do we use for monitoring so ford in terms of documentation we built an internal tool so the nap time api is a huge amount of information that's known statically but unfortunately we never invested enough in really exposing that in a really useful interface to our developers and so what we've done is we're actually using graph ql to actually provide the type safety & documentation or client developers there's a tool called graphical and i realize that i probably did not include the slide and graphical in my appendix that allows developers to incrementally build graph to all queries and that's what we're using going forward it's entirely based off of the nap time statically type information but graphical is a more sophisticated client ide than we could ever build ourselves so we're leveraging the community's work there to help make us more productive as for monitoring that's another great question we use a service called data dog we have open source a integration between let's see it's between coda Hales metrics library from Yammer and that you're able to basically use code hell metrics library and we can you can automatically ship those metrics to data dogs so we have an open source integration bridge layer that a number of companies are using it and you're also welcome to use that gives us white box instrumentation we use filters to help basically catch a lot of generic sorts of things so we can see how many concurrent requests are in flight we can see how many 200s 500 etc over time great question another question so the question is how many programmers does Coursera has we have 65 engineers we're actually growing quite aggressively right now we have a lot of work to be doing another question yeah that's a great question so this bind resource is a macro that generates a routing layer that we then use its we're actually using juice multi binding which came in juice 30 or 40 we then have a special hook in our routes file to basically query this multi binder inside juice that we basically then use that to handle the routing so we intercept requests at the top of the routes file and if it can be routed to one of these nap time resources that's handled else it just falls through does that answer your question that's a great question so for this resource right here if the path is / courses dot v1 that's going to hit this resource if it's not then it goes to a different resource and so you can basically bind a bunch of resources they're all non overlapping paths great question question in the ok so the question is is nap time a replacement of the controllers or a wrapper on the controller and the answer is nap time is a replacement for the controllers developers don't write controllers anymore they instead write nap time resources and we hook in play as a powerful general web framework and they provide a lot of amazing hooks to be able to customize play as you need it and we take advantage of that so that we basically intercept the requests and handle the routing in that manner so the question is is this a wrapper on a play app or otherwise this is just a library that fits inside a play so all of our services are actually placed services they just happen to intercept the requests wrap them through nap time and generate standard play responses underneath the hood but developers have operated a higher level of abstraction that's easier to test have more specific types and has all of the static documentation that were able to glean automatically question in the back I saw first thank you so I'm going to take that in the reverse order the first question is how do we measure the productivity gain and you may have to remind me of the first question which I'm gonna answer second so how do we measure the / tippity game the first is we actually just looked at the source code we're able to see that we're able to produce more types more efficiently so Coursera is a platform is very very broad we're able to build more features each each nap time resource they're all very because they're all very strict they'll follow a very specific schema we can actually tell how many types in our ontology we're able to add and before nap time we're adding them very slowly and afterwards able to add them more quickly this means we can add more and more features more and more quickly additionally we also have the quant this is our quantitative feedback or qualitative feedback is of course that developers are happy with nap time and that they just use nap time instead so that's sort of how we've how we've measured it and we feel we feel pretty confident that this is incredibly well optimized for Sarah's needs for scale on Coursera scale dimensions the first question is so we use a number of advanced language features what happens when they fail do developers just tear their hair out or get very confused and that's a very good question it's hard to basically get it exactly right in general most of these are actually fixing issues at compile time or catching issues at compile time there's actually relatively 0 for the most part 0 run time overhead for any of these sorts of things like macros run entirely at compile time and the way we've structured it you know we avoid all exceptions everywhere and if user code generates exceptions we catch them and log them appropriately the macros if the macros fail in the invocation we we try very hard to actually that's a bug in the macro if they fail and because macros give you enough power to write custom error messages in the compiler that show up with exactly the right line numbers when you actually have the inconsistency that I illustrated here where you have an int when you actually expected your ID type to be a string we're actually able to and in your ID it shows up exactly as a red underline right here saying expected a type string instead you gave me an int for the type classes and implicit suire able to take advantage of the annotation add implicit not found to give custom error messages and help developers do the right thing path dependent types of uses very sparingly we use it very carefully to ensure that it's the right thing and the right type is being used in consistency across a number of different nap time resources and for the most part we haven't had too many problems with that just because we use it for just one very careful area where no other feature would do and and but it's a very straightforward use case then when you have this tool in your language toolbox to build these advanced libraries so we do different things in general the most important thing because all this is at compile time give good compiler error messages with the right line in the source code where things things end up breaking and that and that helps a lot all right if you have any further questions I'll be here for a few more minutes afterwards but I think we're out of time again thank you very much for listening to my talk today you