Scale By The Bay 2019: Themba Fletcher, Runtime Types at Crunchbase

Scale By The Bay 2019: Themba Fletcher, Runtime Types at Crunchbase

Recording: Scale By The Bay 2019: Themba Fletcher, Runtime Types at Crunchbase

[Music] hi everyone so this is one time typeset CrunchBase I'm Timbo Fletcher I manage the core platform team in the crunch base and I'm here today to talk about metadata driven platforms but but not just yet so first I'd like to start with a question and the question is about the underlying nature of our data what's the best way to think about it if we're gonna build a bunch of services on top of it is it it well is it a generic graph or is it a collection of typed relations and of course this is an arbitrary choice but we needed to get some hmm some code written and we need to pick something up front and so it's awfully tempting to choose to think about these as typed relations and why is that well it's because we love strong static types we love our compile time safety net we love it so much that we built our whole platform on Scala and we're really proud of this bias but when you design a platform you're dealing with the interface between systems and suddenly very rudely you're back in the land of runtime type checking and it kind of hurts right someone sends you garbage and you have no choice you just you throw an exception I mean I mean you return a 400 error so yeah or even worse this unpossible garbage is on a topic and it goes to the dead letter Q and someone's gonna get right on that right alright we're gonna handle this so what do we do there's a ton of type systems out there for platforms right open API Avro thrift protobuf and we pick type relations up front and there's a bunch of ways to impose these type Safety's on platforms and and so you pick one or you pick two and perhaps that helps and now you've got types between your systems and everyone knows that a person is a person and it's not a company and it's not an event and it's not an investor and we use code generators right we can create case classes from our type specs and we push those types back all the way into compile time for our services and it's really nice and and you put a little translation layer in there and what we call that an anti corruption layer in in domain-driven design you put that nice translation layer in and your business model is separate from its wire representation and then this is really good like this is a nice strong system great so this is what we're gonna build right CQRS event sourcing beautiful simple right commands flow into this system reads they're faster than a fast right everything's optimized searches are pre-optimized that nearly trivial and we wanted to maximize type safety so we brought in Avro and we brought it up in API we've brought an Avro for the for the Kafka side of things cuz it's nice and tight and an open API has great documentation so and we like to be clever so we ran a code generator across the API spec and now we have case classes for all the business objects on this item alright our responses are fully specked out we know what a company looks like because it's in this back-end and get this we got a surprise boost because we defined our query language in this API spec as well so now the typescript code gets a generated client and we know exactly how to search the site without ever sending garbage to the backend so we got best-in-class type safety for platforms right well at least the green areas are covered client and the client side coding experience it's really nice hang on so so we're gonna derived field to the person he forgot to the full name on their right it's just first name plus space plus last name trivial alright we're gonna change this back we're gonna update the t-norm engine we're on the migrations and the d norm storage it the read service update dun dun dun oh we just got a lot of accepted form the Donora engine as a rookie mistake right I forgot to run the migrations before we yeah okay so update this back then run the migrations update the new norm engine update the read service and then the no sweat right and then 5,000 bad request errors from the read service in the last 10 minutes because CD some completely unrelated feature the clan cut the current got deployed before we finished the migration and so next time we'll run the migrations then we'll update the read service and the Dean or mansion in any order finally we can update this back redeploy the jas code but but we need the spec first to update the read service because we're typed what do you mean product wants another field on companies I haven't finished duck to do this yet so this is kind of awful this is not even a big platform and we're already writing specs about rollout order and managing dependencies between systems I thought types are supposed to save us from all this and then there's more here than just getting stuff out in the right order we want to make sure that the right indices are built and we want them out and deployed before someone's phone goes off because the 95th percentile on the read service is looking weird again and what about the API spec it's kind of getting bloated right we have 34 types on CrunchBase and not like business objects like users or subscriptions or something no we have 34 distinct entities so that we store in the graph so that's a big spec also I lied all this wheel I lied we never figured out how to specify search queries individually for these entity types in our API spec and we never tried because we didn't build any any of this stuff this the product of an engineering friction in something like this I would have killed it before we ever went live or the first piece of it so so instead what we built our core platform on was this we remove the type specification for graph entities from our API spec completely and from Avro it's gone instead of having to tell ya instead of having compiled time safety on things like companies on people and on with investors we take that safety against the concept of a generic entity and its type definition and this thing right there in the middle that metadata serves is called an entity definition any of you def on our platform and so by doing this we take our type safety and we extend it out all to all the areas that are now in blue so what's metadata let's dive in so I did it's hard it's just a collection of entity deaths and a collection of field types entity deaths tell us what fields and relationships are defined on an entity what types of fields were dealing with and what the cardinality of any given relationship is so forth entity definitions are things like organization job person field types that's like string UUID identifier date and then fields a little like the name on a person the start date on a job and then relationships well they just tie them all back together we're in graph land so through all this we've managed abstract away from the API all the details of the D normalizations that we're doing if you do on a search on crunchbase now you do it in terms of the collections available to you and metadata so collections are the plural form of entity definitions so this is like companies but they're also defined by relationships companies employees and your query parameters these are fields from the target and ad definition and then you'll need an operator get that from the field type and then oh yeah your argument and maybe you should make sure that that matches the field type as well so we can do a nice comparison and and that's that's that you know the command language is is nice its graph oriented the query language gets to be more pure - its graph oriented now instead of companies people it's typesafe all the way down but our update problem from before or we couldn't figure out how to roll a really realistic EEMA change out and all the associated code in the right order without a multi-step document well now you just changed the metadata and that's it you're done I mean it's it's event sourced so you're probably gonna have to reflow but but you are done and that's an automated system that's that's that's not a spec I have - right and so the other fun thing I've done we've taken the graph types and we've made them a request time implementation detail all these services in the core loop and this includes the JavaScript in our front-end code they're all they're all metadata interpreters until the very instant of a request they have literally no idea whether they're gonna be serving out companies and people or puppies and brunches just doesn't matter so how do you change metadata in this system the way we do it is you provide it with entitlements entitlements or roles and really the only difference between an entitlement and a role is that an entitlement has no hierarchy enforced so if you can write to our system you're probably logged in we're gonna have an entitlement for each of those facts this gives us really really fine control over the metadata and the permissions that are derived from it so we use is we use all sorts of other entitlements not just the user facing ones to to alter our permissions so for example if we have a data connector talking to some external API and it's gonna load a whole bunch of stuff into our system we don't want it to run amok and destroy everything so when we program it we give it an entitlement just for its use and it can only see the metadata it's allowed to see you know can only alter the things that can alter that gives us a little bit of a little bit of extra safety and type errors so those are features now right remember that the platform exception I said was a bad request or the bad request I said it was a platform exception well it's still possible but no when you get a metadata driven permission error we're going to return a set of entitlements in your in your response body and are you going to take those responses those the entitlement side of that response you're gonna hand them to billing service and it's going to give you a list of products that you can buy to get your metadata into a shape that would allow you to do the thing the origin we asked to so so what's that did we just do we just do a type system driven upsell or or maybe I'm just pushing the analogy and maybe a little fresher than it needs to go so let's break the analogy even further down because we put all kinds of things in metadata so we put validations in and I'd really like to stand here and claim that validations are refinement types but okay we have display instructions themes layouts labels all those things we have application limits the maximum number of rows and a search result that sort of thing those go in metadata and query templates are really cool so if you if you go to crunchbase and you see something like the number of funding rounds on a company it's a link that's a link to a piece of aggregate data that's stored on the company record but when you click it there's a there is a search defined in the field definition so you just land right back into search and this lets you have a really nice beautiful product loop where you click on something on a profile page you land on search you find a new profile back and forth back and forth and it's all just sort of a a side effect of having metadata so drawbacks of course right one that immediately comes to mind his field types are really really hard to modify him this is like if you were in Postgres and you decided you wanted to update the way a fire car works so we don't do that you can of course add new field types but an update is just pretty much out of the question the next ones may be more of a social concern there's a learning curve to working with this stuff this isn't I think standard practice across the industry so you need some time after onboarding to really develop an instinct for what goes in metadata what doesn't what behaviors need to be tied there an authoring it is not not always trivial we have we've built a lot of guardrails a lot of extra tests all kinds of things to to improve the authorship experience but we can always use more tools so then finally and this is less of a drawback in practice than it was just a real surprise for us this actually impacts a be testing quite a lot so we thought we thought when we did a/b testing that meta data service would be the source of it it would just serve the flags turns out that that doesn't work at all what we actually need is both a and B in the metadata and then then when you push that you take the flag in the client up code and it makes the decision about which piece to use works well but it was a surprise so did we achieve the goal we set out to achieve did we build something that was both agile and safe in the first few months immediately following the launch of our platform our tiny little data engineering team I think it was three and a half people at the time integrated with eight distinct data partners deepening our profile experience with four pay data and they did this by authoring metadata by writing a connector by dropping payloads on a topic that's it no platform code changes no migrations nothing and that feels like a win the second one is really my favorite though it's it's not at all uncommon for me to come into work on a Monday morning and find a pull request directly from a product manager against metadata service they adjust their labels they rearrange the content on profile pages they adjust query templates and all I have to do is approve the PR I really like that then both of these hint to me we may have found a sweet spot on which to balance this type safety against product agility so we're getting to the end what's next we're looking at multi-tenancy and the possibility of truly private data on our graph and we're not sure how we're gonna build this yet but when we do we're gonna have to find new ways to express these concepts and we'll we'll probably do them through metadata but more immediately my biggest concern is how we scale this concept across a slightly larger engineering organization we've got 30 engineers at crunch base we're going to ad we're going beyond and there's gonna be no shortage of new voices clamoring for new features pushing this system in all kinds of directions and so finding ways to create to creatively harness this while keeping it true to its original design goals is gonna be a really fun challenge and that's it thank you of course of course I have to mention that we're hiring so if you'd like to come talk to me at the booth that's great I'd love it well that was awesome there's soup equations because there's a really cool concept in implementation anyone thank you Samba that's a great great talk and so I am just curious how like originally the decision to use all this some of this tech scholar Kafka maybe you can talk a little bit how this came about the meditator part or the having the consciousness of the tech stack so the tech stack is not my design this is not my architecture and so what I was really trying to do is focus on the meditate inside of this thing I think you take some you take some constraints on upfront you say I want to do I want to build a reactive system because I think it's better and I think you say I want to be type safe because I think it's better and does that answer your question or no yeah any other questions yeah thank you for your talk I was curious can you drill down a little bit into the the metadata service and what that's built on oh sure you know what components are there sure so it's a NOC IH GTP service it has no backing store that's actually one of the neatest things about it so metadata is it's completely represented in Hakata because we really really really did not want anything to do with XML so so it's ho con because the concept of metadata has there's there's an inheritance in it there's mix-ins how kind it's kind of nice right you can just mix things in or you want them and we can do both you can you can inherit from things by by mixing in and you can also bring bring everything back together you know and solve the diamond problem that way yeah so it's just it's other than that it's a very simple little akka HTTP service built on it loads its its configuration at runtime it takes entitlements off the wire calculates the permissions at the very last second and serves a response serves a big bag of JSON anyone else how are you validating and testing that the metadata that you enter is correct identically so the the one piece of work we did have to do in those first few forensic months after we launched this platform was account for the fact that metadata was no longer being authored by the author of the service it was being authored by by strangers and the biggest problem we had were referential integrity errors so we had to go back through metadata every and every point every reference we make to something else we have to make sure that it's clean and if we're not exposing the entire schema to every requester so it's so it's tough you know you you you take a set of permissions you see what's in the metadata and then you go walk over it several times looking checking for every reference making sure it's that all the right things were filtered that how do I know when I enter the metadata that what I'm putting in there is actually gonna work to do what I want well no we don't know if it does if it's gonna do what you want we can assert that it's referents self that is consistent but you can't actually prove that it does what it wants that so that's that's where you'll write you'll write an integration test against the code to say okay I expect this piece of content in this location I expect this kind of thing because we can't you can't at the service know what the intent is from the product level anywhere else well thank you so much yeah thank you guys very much it's been an honor [Music]