Scale By The Bay 2021 : Anshuman Nayak, Scaling the Disney Streaming Offer Management Platform...
a very good day to all of you uh um as sergey said i'm a principal engineer with the team at uh disney streaming uh services which is the uh streaming business for uh the walt baseball company uh today uh we'll talk about uh uh i'll give an overview of uh of growth uh growth engineering what exactly does the responsibility of growth engineering in a streaming business and uh why we made some a lot of investments around like like looking at the code and refactoring it for scale and as and then what are the challenges we faced as part of that and then you know talk about some patterns that we we learned some of those exercises that we did right hopefully share some learnings and takeaways so um the user growth right so uh we we the disney streaming uh is a pretty uh recent player in the streaming business right so we launched about about two years back and uh and we when we did that at that time uh we were uh projecting between uh 60 to 90 million uh subs in five years right from 2019 and and that was mimicking the growth of some of the other streaming services right out there right so so that's that's where we were right but uh happy to say or you know we were our projections were completely wrong right and we were like presently surprised and very quite humbled i would say by the actual numbers right so on launch day uh we were we were at 10 million sign ups right and and then very quickly you know across the various quarters we we grew pretty fast from 28 50 to to now almost 116 million subs as as per the last uh earnings right so so that's like a quite a huge growth right and and that growth story led us to kind of look back into some of our services and and figure out what are the uh things we need to do uh to handle the revised projections right that we have right and and and that's that's basically like what was the driving force for some of the uh changes that we had right so uh when we are talking about uh especially growth engineering uh uh uh it's not about not only but most of the challenges are uh are not only limited to subscriber growth right like as the number of uh subscribers uh uh new subscribers uh are there for your platform but it's also as we as we uh like launch in various new geographies right of the world uh how does uh a lot of uh you know the challenges that are there when you launch like new countries right a lot of challenges around currencies uh you know local currencies there are local partners right who who understand that that region well and and you can use them to sell your product right and what does it mean to do that they're local payment providers right that you have and how what does it mean to integrate with them so that you have a good sign of experience [Music] there are a lot of you know a lot of challenges it's not a simple push button so what does that mean uh right so so those are some other discussions so so what exactly is you know at a very high level you know what exactly is growth engineering right growth engineering in in disney streaming it's responsible for uh customer acquisition and and retention right what does that mean right customer acquisition is how do you uh how do you get uh you know how do you show the right uh offers right or the right products or right content to the to the to the users so that they can sign up and become paid uh paid subscribers and that's uh that's like a free key vehicle of growth it's not only new users it's also existing users how do you show them new uh how do you upsell to them right how do you get them to buy new new products right and and that's that's where the offer domain is responsible and then you have we have get a subscription business right so uh what what does it mean to that you have different people different uh users who are in the different life cycles of their products right they have bought the products with various offers and and in various stages right so what how do you manage their subscriptions and how do you find the right uh invoice for them right based on the invoice date uh and and then how do you help them move the money around right like actually charge uh the the payment methods get the money uh and and doing all of that uh like like repeatedly you know for for almost 100 million plus users on a on a monthly cadence or whatever cadence it makes uh you know you are signed up so all of this is quite challenging right and doing it for scale so that you know there is predictability and there is good uh you know customer experience right so that we can retain our customers right is is is what growth engineering is is responsible right also like in this particular slide at the at the left uh you have all these various uh clients and and you know devices right which where you can watch uh disney parks right so it talks uh through the edge layer to a lot of these uh growth engineering backend services and and and at the at the right uh you have a lot of these uh third-party uh partners at the integration right so sometimes we use uh as we go to various countries we use some of these partners to help us sell our products right and these could be the the apple stores and the roku stores of the world or they could be local partners like comcast and verizon or canal in in the in the various countries right and and and in in in some cases we are the source of like what products to sell what are the prices to charge for the products and in some cases you know these partners of the source are like how have the users signed up with these partners what are the billing terms you know for these partners right and and both of these systems need to be in sync right so that you can enjoy a subscription you bought in your app store in in other areas too right and the activation domain helps with all of that so that's that domain also talks to the edge layer so so uh so at a high level that's what the growth engineering does at uh you know at disney stream right in the rest of the slide we'll talk about offer and offer domain and what are the you know how does one of the challenges and how we solve that right so how does offer uh help with growth so as i said uh like you know the one of the key vehicles for increasing our subscribers is to show uh you know uh content and offers which are contextual to users right something which will make you sign up to business uh is is might be different from what excites you right so so how do you find the best offers for our users right so that's where offer domain uh helps uh drive that right so it's a key vehicle for growth the other dimension is as i said geographies right as you as you launch in different geographies uh how do you uh you know how the lot of key metadata that the offer domain manages to help with that and and that's again how of a domain becomes a key vehicle for growth right it's a subscription business so you know what i in in most subscription business you know after some time uh you would end up increasing the price for subscription right so and and doing that uh you know for uh for like 100 million plus subscribers uh is has its challenges right how do you do that in a in a predictable way right and and the offer domain uh manages the the price is responsible for pricing so there's a lot of um involvement that you know for for the growth right growth of revenue growth in this case so what are the key challenges right what what did we what are the challenges for the offer domain and then of course the next slide will be like how we uh what are the patterns we we discovered right so what are the challenges right it starts from uh like the basic right like the products what are we trying to sell right so when we launched about two years back we had only one product disney plus right that's a that's the only product we said two years now we have four major streaming platforms we have disney plus we have hulu we have espn plus for the sports offense and we have star plus then we have a lot of these on-demand pay-per-view events right which people buy and then all of these theatrical releases right which which go through premiere access so a lot of these products have grown uh like in in two years and and they are supposed to expect it to go even further so how how do you uh like you know scale your or your systems for for each of these offers or products have their own unique needs their own unique economics so how do you build your system uh to scale right to to for for a huge influx of these products these products have their own uh they don't not necessarily only give you entitlement to watch something like a content you could be you know a feature like hey whether you can watch ads or do not watch ads uh how many streams right we have can you record some of these can you download some of these there's so many various uh you know other dimensions to some of these product effects so that's like how do you do that in a scalable way the next is uh as you go for uh multiple geographies uh how do you um you have unique requirements for all of these your graphics right currency of course is very okay every year obviously we have local currencies but then you have to you know if you have to show these content and offers the user so you need to it needs to be localized right how do you manage localization you could have different uh payment methods right for for each of these countries you might have different authentication requirements right like like europe has strong authentication techniques right how do you understand all of these how do you as i said you also have uh different local partners right for for a country and you might want to integrate with those partners so that some of the uh so that you can you can can reach your customers in an easy way right and then payment methods right so all of these managing all of these or the metadata around all of these is something that uh yeah offer domain hold so how do you build a system so that each time you launch or you integrate with a new partner launch to your new country you do the bare minimum stuff required and then you know you can scale faster that's another key challenge for the offer domain next is you know uh like some of these uh offerings that like uh like offers that we have uh very often business comes up with uh very uh like offers which which are you know which will help our users sign in so like how do you identify uh like you know groups or cohorts of users right how do you characteristic of users right and then offers which you know which these users can sign up for so how do you do all of that uh you know in a in a way that uh you know they it can roll out faster right so you don't want to have engineering involved in you know every time when uh you know you you business comes up with a new way or the new uh new offer or you know uh and so how do you scale around that right that's another uh important challenge for the offer today right reducing time to business for for these offers and then of course as offer domain is manager's price so you know uh the price of the product so when a price change happens uh you know you often do that in a rolling way or in a rolling fashion like do this country first other country next even within a country you might identify sections of users for which you might not want to uh increase the price keep them in the old price per sub for whatever reasons right so how do you you know do that in a in a scalable way right without uh you know ah for as i said for the 100 million plus subscribers and growing how do you do that in a scalable way so those are you know uh the some of the key challenges and again you have uh a flow called redemptions right uh like like lot of time when you say buy a device like you know or a tv you might get like a code like a sign up code which says you know go redeem this and you know you get couple of months of disney plus free right how do you manage the life cycle for for all of these uh codes a lot of these times these codes are you know could be um could uh are amenable for fraud like so how do you detect fraud and you know all of these rights all of the life cycle around redemption codes [Music] some of our services to see how we can uh you know address some of these challenges in a in a scalable way right and that's we did that and and that exercise and i think i guess in the rest of the slides uh uh in this talk we wanted to present you know what are the key key learnings that we have right around that the first one is uh it's kind of yeah we kind of yeah we understand right domain separation right we should ensure that our you know our micro services are aligned to the main boundaries right but often what happens is you know as we are as the business evolves uh we start you know we realize that we start leaking uh our data models right which are which are for a domain right we somehow sometimes leak it to the interface and and when we do that you know anytime you want to refactor or change your service now you have so many dependents dependencies and and that would be a bad pattern right and again you know you have uh you know domain boundaries which start leaking to each other so one thing we did is you know we took a step back and we said for the growth domain right let's figure out uh what are the different domain boundaries which make sense you know what what are what are the key responsibilities for each domain right for example in offer domain you know it was about uh we are responsible for selling or or figuring out what products you sell so that's product domain right we are responsible for the offers offers our economics right with these products when you sell them what are the economics about it right so that would be uh offered economics to me right and then we we would have domains around which users are eligible for these for these uh offers so that's the eligibility domain and so on right so figure out the right domain boundaries uh so that uh you know so that you are yeah you can scale well right often what we have seen is that you know the business keeps changing uh you know uh especially in a fast uh fast market but if you have your domains well right if your domains are they are a little bit more resilient right so you can always build uh you know build business use cases around your domains right and and that would be that's why it's worth trying to always take a look back step back like every once in a while to ensure that you are your domains are still pretty aligned right the other pattern was around you know we we have all of these uh you know different micro services right and like in in in the growth domain you have a lot of these micro services now they have they are they're the source of truth for some kind of data right but for someone who is outside the domain like all the clients in this case right who need data for from us for them to figure out which is the source of truth and how the data related to each other it's like it can be quite complex right so that's where normally have you you need a kind of a orchestration pattern which can which can help uh you know keep the abstraction right and and we looked around and we found graphql to be look pretty uh something which we were quite interested with right and how graphql can can abstract out you know some of these uh like data it basically can abstract our data and the relationship between data for the clients right and and it does that while keeping your micro services loosely coupled which is what what we wanted was one of the key requirements for for us right also you know uh it it provides like you know a lot of time you have all these devices that you have and each of these devices have their own uh like sign up flows or you know flows that they have built based on how much you know payload they can you know they can manage in those devices so so it's a lot of time the amount of data that is required is also different right so so graphql through a query based uh approach uh to getting data helps in that uh process right so again i think you know uh the key pattern we we really wanted to emphasize is yeah and try to have uh loosely coupled micro services uh you need orchestration to to you know to help the other clients and domains you know like understand your data understand the relationship with your data and and graphql uh help us the next pattern uh was around configuration this is a little bit uh you know we found was uh a little bit overlooked in in many cases a lot of times when you when you have your own domain like in the growth domain you have many domain specific configurations right uh and and you know and and when you have so many of these micro services it it could be you know a pattern where you start making local copy of your configurations and and when you do that you you pretty much start losing uh like what is the source of truth for simple things even like you know what are the payment methods for the country right how do you uh you know how do you figure some of these things out a lot of times you know when we uh we like sometimes we also see that when you go to a country all these different payment methods that you have even the amount of decimal places right you can charge a payment method like uh that can dip that if you have more or less it can it can lead to your payments being on not not authorized and uh from a payment method which is local to a country so how you need a central source of growth for a lot of uh you know of of these configurations that's where you know manage your configurations well don't make local copies uh if if possible right some of the other advantages if you make them uh like you know central you can schedule some of these right so a lot of times these configurations have to change values uh in like for example the price changes right a lot of time price change happens you know for whatever reason in the middle of the night and that's where a lot of these fights price changes happen and you don't want to be like awake and doing a lot of these changes right at that r right so you you want to schedule your changes right and and this configuration changes right so you can do all of that in a smart your configurations can immediately become smart right and you can as when this value changes you can have listeners like your micro services which care they can be listeners to the configuration and and you know they can get a like a call back when something changes and then they can interact accordingly right so so build smarter configurations right you might also want to scope your configurations like a lot of times you have configurations which have like a global value but then a specific microservice might want to scope it to something else right for whatever reasons right so a lot of these smartness can be built uh if you have uh like a central uh configuration that's again another key learning like if you have to scale uh you know try to uh not have local copies of your configuration one of the uh challenges we mentioned for offer was uh you know enabling our business to be creative right in the kind of offers they launched like like for example you know a simple offer a relatively simple offer could be like if you are us quote unquote student in brazil with the whole status for like you know three months of d plus disney plus and you are still watching let's say start plus then maybe give you other incentive like give extra like some extra percentage off so that you can join the service right that could be uh offer right so there are two parts of this one is defining what are the economics of the offer and and when does the offer you know the the values change the second is identifying the cohort right or the group of users who you want to target in this case like you have to be a student so you know figure out whether you can be a student in in brazil so figure out the country uh you know figure out your account status figure out your watch history and you know and and then build a you know build offer for you right so in the initial days a lot of times engineering is to be involved in in doing a lot of these right and and that used to not scale right that never scale right we want our business to be very creative and and as we launch to different countries some of these can be very local to the region so that's where you know the idea was to get into configuration driven right it's you have to figure out how to ident how someone can uh you know specify you know the the cohorts or the groups and then how do you specify the economics uh you know in in an abstract configuration way right and then the role of offer becomes you know when when someone uh like is lance on your on your platform you can you can figure out like you can get the characteristics of the user and then evaluate it to see if which cohort or which group they belong to right so that becomes like a more dynamic way a scalable way right to approach this problem so again you know always you know the the key learning is you know try to try to move into configuration driven approach uh where possible right that would uh that would help you uh scale a lot and that's that's the key learning we had once we did that a lot of times you know uh we have our business users come in and and set up these things and and then you know engineering doesn't have to get involved the the pattern out here i call it transitions but basically it's it's mostly also to say that you know our entities try to keep them uh immutable when possible right uh there's like in the early days like when we had uh like uh in this case uh offer right the offer entity that we had a lot of time people used to go like whenever something changes right we used to go like a price change happens right people can go and change uh these entities right and then and that's how it worked right but as as we realized that as we as we as we have a lot of these offers as we have like 100 million plus users and their records point to some of these entities changing uh you know making these entities immutable is is important because that leads to you know traceability that leads to you know again scheduling you can schedule some of the changes uh i know so how do you you know so that would help in in the scale for the business right so when you have your entities which are immutable you know now then the question is how do you move your user records from one entity to the other entity and and that's where transitions help right where you can now programmatically do all of this you can say that okay if you have these characteristics of the users and that could be country you know date uh you know you uh or or their their paying pattern you can you can identify those users and move them you know in a programmatically so that when the when the billing cycle comes you know every month you know how much to you know where the users are right so so those that's what we call transitions but in a nutshell it's saying that keep your entities immutable and then find a way a programmatic way to move your users across across some of these entities right so key takeaways uh again uh to summarize create micro services right always go back don't leak your data models to your boundaries to your interface boundaries that will uh that will cause issues have orchestration federated graphql work for us uh you know to to get data uh you know uh out uh or you know it's required manage your configuration centrally uh and that has a lot of advantages uh make your entities immutable and and try to have configurations as much as possible thank you thank you very much you