Scale By The Bay 2020: Steve Cosenza, Rebuilding Twitter’s public API

Scale By The Bay 2020: Steve Cosenza, Rebuilding Twitter’s public API

Recording: Scale By The Bay 2020: Steve Cosenza, Rebuilding Twitter’s public API

[Music] so how many of you have decomposed a monolith into microservices and how many of you have created a multi-tenant microservice to eliminate the need for other microservices over the years twitter's learned a lot in this space and we now have an api plan that we hope will scale well into the future so today i'm going to walk through kind of a brief history of the twitter apis followed by a discussion on the goals and design of our new public api platform and then we'll finish with a real intent example of how engineers build api endpoints using open api spec graphql and scala so first let's start with some history our journey starts 14 years ago in 2006 when twitter launched our first public http api which was served from a single ruby on rails monolith which later became known as the monorail by 2010 we had one of the largest ruby on rail systems in the world but given its growing and complex single code base was becoming increasingly difficult to make changes and it was almost impossible to paralyze work among our engineering teams it was at this point time for migration and in 2012 we completed migrating our public api off of the rails monorail off of the rails monorail and onto numerous thrift jvm microservices at this time fronted by a single hdp jvm microservice named woodstart now woodstar was instrumental in getting almost all twitter traffic onto the jvm but in time it became its own smaller monolith which was becoming increasingly difficult to change and operate so the drums started beating once again and it was time for yet another migration and in 2014 we completed the second migration from a single woodstar api service to a set of 14 http microservices running on an internal jvm framework named macau now this microservice microservices approach enabled really fast development speeds at first but it also resulted in a scattered and disjointed twitter api as independent teams designed and built endpoints really quickly for their specific use cases but they didn't coordinate so fast forward to 2017 when the twitter app started using some new internal only rest and graphql apis and having internal only rest and graphq graphql apis helped our twitter app teams move fast and iterate quickly without having to rely exclusively on the public api which at this point was time consuming to change sorry about the extra noise hold on all right so in early 2019 we started to plan for the next major version of the twitter public api and we knew we needed a new architecture to address the public api's slow iteration speeds and continued fragmentation which brings us to this year when our new architecture started powering the public api platform which now hosts our first api v2 endpoints all right so with that brief history behind us let's now look at the goals and architecture behind our new public api platform so for our new platform we knew we needed a new architecture that could more easily scale with a large number of api endpoints needed to serve current and future functionality and as part of this design process we drafted the following goals our abstraction goal is to enable twitter engineers to build on the api and focus on just the querying and mutating of the data they care about without needing to worry about the infrastructure and operations of running a production hdb service our ownership goals seek to contain core and common api logic in a single place owned by a single api platform team while also allowing non-common api components to be authored and owned by different teams and our goal of consistency is to provide a consistent experience for external developers by relying on our api design principles to reinforce uniformity so with the above goals in mind we built a common platform to host all of our new twitter api endpoints and to operate this multi-tenant platform at scale we knew we had to minimize endpoint specific business logic otherwise the system would quickly become yet another unmaintainable monolith so a powerful data access layer that emphasized declarative code over imperative code was really crucial to the strategy and as we discussed earlier in 2017 our twitter app started to move to an internal only graphql api and the momentum to move to graphql was really building throughout 2019 when migrations started to happen across the company so it was really perfect timing and our team followed suit as we realized that the data querying needs of the public twitter api are really similar to the needs of our twitter and mobile desktop clients put another way twitter clients query for data and render uis while the public twitter api queries for data and renders json a bonus from consolidating or data querying through a single interface is that the public tutor api can now easily deliver new twitter features by querying for graphql data already being used directly by our consumer apps all right so the graphql-based platform approach decided we needed a multi-tenant way now for different teams to build and contribute to the overall twitter api and to facilitate this we designed several plugable platform components and this talk will focus on resource fields and selections resource fields are used to create the core resources in our systems for example tweet resources and user resources and selection components are used to define how to find resources for example tweet lookup by a date using these components teams can independently own and contribute different parts of the overall twitter api while still returning uniform representations in responses all right so at this point in the story you may be curious where endpoint specific business logic actually lives and we offer two options here when an endpoint's business logic can be represented in strata ql which is the language used by twitter's internal data catalog system then we only need to write a function in a strato-hosted managed column without requiring a separate service otherwise the business logic is contained in a thrift microservice written in scala and exposed by a thrift strata column in either case the platform provides the common hdp needs for api endpoints so new apis can be released without spinning up a new http service and if an endpoint can be constructed by querying for already existing graphql data or if an endpoint business logic can be implemented in strata ql then we can bypass almost all service owning responsibilities for that api so now that i've discussed the history of the twitter apis and high level goals and design of the platform let's walk through an n10 example the requirements from our product manager are as follows we must handle an http get requests to slash two tweets that contains a single tweet id in the path if the request is ill-formed return a 400 otherwise return an http 200 with a json response that has a default id tweet field and a default text field we must also handle an http get request to slash to tweet search recent that accepts a query max results and next token prim again if the request is ill formed return of 400 otherwise return to 200 containing a list of tweets and the next token so that customers can paginate and similar to tweet by id we need to return the same standard tweet format that's used elsewhere in the twitter api our requirements will need us to create a tweet resource which has an id and a text field that are always returned so on screen you can see where both of these resource fields would be defined in our common config directory let's first define the tweet id field on the top you could see the contents of a fragment.graphql file and this fragment is used to retrieve all the tweet data needed to render just this field in this case we query for the ide string field and at the bottom of the screen you can see the contents of the field.scala file which is used to configure various aspects of the field the owning team is used to determine where to send automated pages when this field is experiencing problems always include being set to true makes this the default field which is always returned and then we specify that this field should be returned in all major and minor versions of this api finally we use a special path based renderer which lets us declaratively specify how to render this field without the need for imperative code now when imperative code is needed for rendering or error handling it will exist in this scala file but the platform has a goal to limit the need for imperative code wherever possible next let's look at how we define the tweet.text field since the graphql and scala files will look almost identical to the tweet.id field let's focus this uh this slide on the two additional files that are present the first file is a project file which allows different directories in twitter's monorepo to be owned by different teams in this case we include a file which indicates that tweet text is owned by the api vnex developers group and fabricator which is the system twitter performs its code reviews by specifying this project level ownership fabricator will ensure that all changes to this directory have a ship it from at least one member of the owning team so that's one way we're able to uh distribute ownership across the platform second file at the bottom is the slows.json file where the field owner can define an slo for retrieving and rendering this individual field an slo being a service level objective um so in that slo we define a 30-day service level objective and in this case we say that the successor needs to be 99 now this file isn't absolutely required but if it is missing we'll provide a default file um for the platform okay next up sorry about that okay so at this point we've defined two fields in a tweet at the top left you could see the graphql fragment for retrieving the id string and below you can see the graphql fragment for returning the full text field now part of the platform's build process is to read through the entire config directory and then generate the fragment on the left now we can now issue a single graphql query for that platform on the right instead of having to request two separate graphql queries and this is kind of a key to using fragments as our rendering as our way of doing rendering reuse and if you're familiar with react our setup is similar to react relay combines individual ui fragments into larger fragments depending on which ui components are included in a composite ui page so next up let's look at how we configure selections recall that selections are how resources are found in our apis and for this example we'll implement two selections tweet by id and tweet by search okay first let's define a selection for looking up a tweet by its id on the left of your screen you could see the openapi.yaml file that defines where this selection will be exposed in our twitter api the version prefix variable is a placeholder that the platform fills in for whatever major version of our api is currently enabled and the id variable is a path parameter which is a required string that must conform to the specified objects in the top right you can see the contents of the selection.graphql file which specifies a query for retrieving tweets and note that the required string param in the graphql query corresponds to the required path parameter defined in the openapi.yaml file we then call the tweetbyrestudy field in twitter's graphql schema which returns a tweet type and then specify that we wish a platform tweet to be returned recall that the platform tweet is composed of all the data that's needed to render a tweet finally we have the selection.scala file which is similar to route that scala which we just looked at so we could skip that here okay putting all the pieces together the top row and screen shows an http request for slash to tweets20 that comes into the platform resulting in a graphql query named tweet by id to be sent with a graphql variable um this graphqv variable sets the id to tweet id20 and then each resource field reads the graphql response and renders its field value resulting in the rendered tweet seen in the below http response next up we need to [Music] create a selection for retrieving tweets by search and similar to tweet lookup by id we define an open api yaml file this time it has additional typed schema validated query params selection.graph is where we search for tweets using a search query and note the place that the platform tweet fragment is in the query that's where where we expect tweets to be returned and as a reminder our graphql schema is based on our stratodata catalog system so in this example when we query for the match tweets it's actually implemented as a thrift service in this case named id hunter which takes a search query num results and a token as an input and returns match tweet ids and a cursor as output finally we have a selection.scala file which has an optional fail hole request method defined and that allows us to create a more detailed http 400 response based on what our thrift search service returns to us putting all the pieces together here if an http request for slash to tweet search recent comes into the platform with a max results value that doesn't validate against the open api spec then no graphql query is issued and we immediately return a foreign response and now let's look at a well-formed hp request this time we issue a graphql request with the graphql variables populated from the http requests query params and then at the bottom we render tweets in the standard expected format with the default id and text fields all right we've successfully built two new api endpoints capable of selecting tweet by id and tweaked by a search query but wait a new requirement comes in and our product managers now want to start exposing tweet metrics through our api this new field will have a separate engineering team working on it and will be optionally returned only if the customer requests it this new field will be named public metrics and on screen you can see how this new field will be configured as we've seen with the tweet id and tweet text fields we first define the graphql fragment to retrieve the data we need to render and then we include a field.scala file notably this time setting always include to false with this new field defined the api platform will now make this field available to any api customer that specifies that they want this optional field now since this new field is optional we'll only want a query for it when an api customer actually requests it to do this let's take a look at how the platform generates the tweet fragment as you can see on the right our api platform adds a graphql include directive on every part of the query associated with an optional field and in this case you could see the enabling of the public metrics field variable will result in these public metric fields being included now some of you may be wondering why we need to use an include directive instead of just including these optional fields in our query when a customer requests them the reason is that twitter's graphql system does not allow arbitrary queries to be run in production instead we utilize persistent queries which require our api platform to submit all queries at build time in exchange for an operation id that is then used in production next let's see what a customer request looks like for this new optional field first up is tweet by id and at the top of the screen you can see an http request requesting our new public metrics field because an optional field is now specified we set an additional graphql variable named public metrics field to true which will enable our graphql include directive to include the additional fields we need to query we could skip over looking at the graphql response and at the bottom you can see the individual requested tweet now has the new public matrix data included all right next up um let's look at how this would work for uh search once again the public metrics field graphql variable is set to true and then below we can see the search results the two default id and text fields along with the optionally requested public metrics field these examples hopefully illustrate where api platform really shines in letting individual developers focus on specific api components in this case adding a new tweet field component the platform then combines all the components together to ensure consistent overall api so now that we've satisfied all the product requirements let's briefly look at how we alert and monitor our new aps all components in our api platform have service level objectives defined and our platform ensures that each of these components gets automated alerts and monitoring dashboards generated for them and on screen you can see what a pagerduty alert looks like for an api component that my team owns and then from this alert our customers can click through to a grafana slo dashboard showing the status of their long-term slo and the various errors scoped down to individual api components seen across the last one hour six hours and three-day burn windows which is what our alerts happen to be based on now i'd love to do an entire top in just the slide but since i lack the time to dive in much deeper i'll quickly mention two things first since each component in our api is owned by a single team we ensure that critical 24 7 pages are only dispatched for problems that a team controls for example if the tweet public metrics field starts failing which causes the entire tweet by id and tweet by search wraps to fail then the field owner will be paged and woken up in the middle of the night but will never wake up the route owner as there would be nothing actionable for the route owner to do secondly for deciding on when to alert an issues we use the multiple burn rate alerts technique from google's excellent book titled the site reliability workbook and i'd highly recommend checking out this book to learn more about this powerful technique in closing thanks for joining me as we walk through a brief history of twitter's apis followed by why we built the api platform and then finished off with a handful of real world api examples for twitter this new api platform is just the start of our journey and our work is far from done notably we have many more existing endpoints to migrate and entirely new public endpoints to build and yes if you're keeping track us migrating existing v11 endpoints to v2 does mean that twitter is currently in the midst of our third major api migration so here's hoping good things really come in threes thank you