SBTB 2015: Mike White, Legacy Modernization from Monolithic PHP to Reactive SOA
Recording: SBTB 2015: Mike White, Legacy Modernization from Monolithic PHP to Reactive SOA
yeah let's hear it for PHP how many people are actually currently dealing with PHP as a kind of modernization is one over there but ruby-on-rails ruby java javascript java visual basic I've seen we have legacy Scala in our in our organization which is too embarrassing to even talk about i'm going to talk about like the good stuff which is which is PHP all right let's go there are two kinds of startups ones that achieved some modest traction on top of a pile of code of which they are vaguely ashamed and the ones that go out of business and I want to talk about the second kind I want to talk about legacy software and and traction on top of the legacies legacy software and the cost of that so I talked about that and I want to do one case study because kind of pressed for time about a PHP app that we did two scallop so just a little bit about me i'm a senior engineer at HootSuite I've got some large number of years of Java my background is in enterprise software modernization and COBOL and Fortran and RPG and mumps and stuff like that too embarrassing java ee which is in a way legacy to more more legacy but now it's it's gala we're going towards Skala powerful language and i'm loving it and and I'm never looking back to that kind of the previous age a little bit about us the only important thing here is that we are seven years old we built our our success upon PHP and that rapid development framework to the point where we've got 11 million users worldwide and 2000 enterprise customers so a lot of traction so the problem with legacy software i like to call it fossilized technical debt because a technical debt you can kind of pay it back right you can kind of spend time on technical debt refactoring and and kind of paying it back and bringing yourself back up to zero and then it kind of it kind of gets older and older but fossilize technical debt or legacy code you're spiraling out of control your your maintenance costs are spiraling it's expensive to host it and deploy it it's really expensive to scale it in the modern SAS universe it's very difficult to test it and when you are in a legacy environment changes themselves incur risk and risk becomes fud and fud turns into innovation paralysis in your organization and as a technologist that makes me really sad because technology should never be the bottleneck of things I mean the bottleneck of your organization's you know takeover of the world should be sales and marketing and and agility but technology should be the easy part it should be easy to do do things with technology and that's why that's why legacy code is is a as a problem and so our legacy problem revolves around around PHP a dynamic a dynamically typed language scores and scores of lines of custom boilerplate do things that should be done in a library massive code base no real modularity sequel statement strewn throughout no no layers or anything just you know the stuff that you would expect from a code base worked on by people just trying to get product out the door and and fill fulfill a market very very low test coverage and with that comes huge huge risks in refactoring of that code PHP everyone's down about PHP but why why does why does everything why did everyone use PHP and a Linux Apache my sequel and PHP stack the lab stack why did everyone use that eight years ago why was it so easy to to spin up a dynamic a dynamic content serving server web server on the internet eight years ago with the lamp stack because of the PHP community the PHP community made that the trivia the trivial way to do things i made the de facto way of getting a dynamic web page onto the internet eight years ago was PHP wouldn't it be great if the de facto way of getting a responsive scalable 69 s a SAS product into the cloud today wouldn't it be great if that solution six years from now was Skala it became scala because i think with scala we're in a we're we're PHP was two years into the the PHP the PHP take over I think we have an incredible opportunity to make skala the de facto the de facto framework to to go forward that was a little digression so why why Skala what was what was my organizations of reasons to go to Skala where our requirements to to go to two scallop well or two are to address some of the the legacy code we were acquiring where they were there were acquisition requirements there was a move towards providing a platform as a service and providing api's there was cost spiraling cost of serving all this stuff in in in AWS we needed something scalable into the future we need something that would scale with our projected user base without you know paying half of our revenue to to amazon we wanted composable microservices we needed integration with our PHP monolith so that we could we could we could gradually go towards microservices and we wanted fault tolerance and partial outages you know when you're when you're at home surfing Netflix and you go to your you go to the ER episode or whatever and it starts at the beginning instead of running at the where you know the 13 minutes in where you left it the last time that's Netflix having a partial outage of their of their microservices architecture the systems that store the bookmarks of your TV shows are down there having you know they're having an outage but you can still you can still watch Netflix you just have to start at the beginning and then kind of fast forward through that was kind of the goal of what we needed we with our with our software as a service we wanted you know during the when the inevitable outages happened we wanted them to not be you know 500 level errors happening to every single user we wanted you know parts of the system going down when when when things inevitably did go down oh yeah and during the migration zero downtime we needed to the the application itself needed to maintain its its enterprise SLA s which was six nines of uptime and that was a requirement for straight from the beginning so how do you how do you modernize legacy code that is serving up thousands and thousands of requests a second to paying customers how do you migrate that away from away from PHP and into in this case scala so to back up our engineering strategy had to change to go from from PHP and maulana the two Skala and SOA so we performed iterative experiments leading to our commitments under decisions we always cycled around building building an experiment measuring the results and learning from that and always doing it in production so we did have we did have a development environment staging environment but the only thing that counted was production running these experiments and running these the these changes in production was the only thing that we focused on and of course our decisions of course scale and akka of course we were very well aligned with the the reactive manifesto it just makes sense for a for a SAS offering we went down two paths we did our own microservices architecture which isn't in this talk and then we modernized another another mana left into the flight play framework which I'll talk about later we scaled our team by hiring Java developers by training PHP developers we partnered with those rad guys at bold radius to do a series of training sessions which worked really really well but we needed to scale our engineering practice and change our engineering practice itself and for that we turn to Conway's law Conway's law says that if the Conway's law says that the design and architecture of your system will follow the communication and team structure of your of your organization so here we have the UI specialists we have the middleware specialists that's the monolith and we have the dba's handling the database and of course that's the that's the the mana left with a you know the the compartmentalised you I middleware and DBA instead of that one way of reorganizing is to is to create cross-functional teams and aligned around vertical slices of the product that align with the micro services that you that you want and so that's what we did these are the HootSuite the HootSuite product offerings right now this is you know present and future and each of these each of these represents a engineering shift like an orthogonal engineering shift to change the way that that we organize ourselves so that's how we did it kind of an abstract organizational way we I'm going to talk about the modernization of our URL shortener alley who needs a URL shortener in 2015 well we do because one of the things that we wanted to offer was a free and automatic URL click and i'll click analytics so every URL that was posted through the the HootSuite product to the various social media networks that we supported was shortened with Ali and and we provided click analytics for free to everybody Ali has a public web UI so you can actually go to the go to the system a public API sorry a public web UI so you can kind of browse to it a public API so that you can kind of use it as a as a software as a service and then a private extended API used by our dashboard product for all URL posting we are kind of the we are the provider of many vanity URL vanity URL shorteners so every time you see like you know typesafe dot es or whatever it is I don't think a type-safe but you know you know what I mean right the the vanity URL shorteners that are out there a lot of those are backed by by ally itself and up until a year and a half ago they were backed by six year old PHP code just some stats a thousand you were a URL redirects per second ten images uploaded per second ali is the second most popular URL shortener behind bit ly that you've never heard of TCO doesn't count because you can't go to TCO and actually it's not independent outside of outside a footer so for ali i kind of want to draw attention to how we found it oh the legacy problem of ali how we scoped and estimated the modernization effort how we executed the modernization and a few skala patterns because there's a skala conference i needed to throw in some skele so each one of those little PHP icons represents an AW instance AWS instance I think there was like 60 or something they were m three larges or C three largest so the you know the relatively expensive ones it was very expensive the snowflake deployment we had to call the guy that knew how to deploy it and get him to you know tore it up or are synced or whatever insanity was involved with with the deployment there were no tests there was it was an old version of PHP which had security holes the business of HootSuite business was looking to monetize the traffic we were we were getting a lot of you know especially on mobile and ends but and for the image landing pages they wanted to monetize that traffic they wanted to kind of you know show ads or whatever and we were like no no we can cannot touch that code it was was total paralysis with this you know this valuable commodity because of legacy software it looked like this so what do we do well on the front end we we changed the HTTP API and and web UI into play of course back-end we wrote a micro service data microservice around around the database to provide an API around you know persistence of URL shortened URLs it was a cross team effort like we said before instead of just having you know platform people and everything it was it was a vertical slice of of this team that was enabled or mobilized to take on this modernization 0 plant 0 plans down times and very few bugs you know I think I think the total downtime during this huge pivot was I think measured in measured in minutes maybe two or three minutes you know measured in like 15 second increments over over a couple of months so not not a lot of downtime considering the magnitude and risk of the project and no huge scoping failures this is what it ended up with so a play on the front-end nine servers and and a data service on the backend another nine service and these are what this is without any optimization we could have gone down to fewer for elastic scaling we could have done something different so how exactly did we modernize it well we went to we looked at development best practices like the the modern development best practices so unit test coverage on every single line integration tests tests on on every every API and every screen environment parody across our utilized environments the centralized dev environments the staging environment and of course production proper monitoring logging and alerting per our best practice through the the rest of the organization and always go to production with the minimalist Viable Product so every time any little vertical slice was done pushed to production and then feature flag that small slice for the for the monolith so remember we have a private api being called by the monolith and many many thousands of times a second for for URL shortening every time there was a single slice well we would we would do an incremental deploy with behind a feature flag into into that slice so we'd be using the old system and the new system at the same time in production while we were while we were developing it so speaking of clear scoping scope analysis became huge when we were staring at you know 500 thousand lines of PHP and in deciding how we were going to modernize it and how much it was going to cost the heart cell right because because management is like you're saying that it's an architectural change a new system that that is exactly the same provides the exact same functionality but it might be a little bit worse because it's new it's relatively new code and it might contain bugs hard sell so how do we sell the number to management well we started with a user story like those like the user stories you see there and then we took a look at a hard look at the deprecation so API versions screens cross-cutting features and deprecated technology so we kind of thought about those two kind of orthogonal those orthogonal things so user facing stories and then deprecated deprecated concept the scope analysis only took a couple of weeks the product owner and architect book both times off on the the deprecation 'he's that were appropriate for them to sign off on so the product owner only saw obviously the product owner only signed off on deprecation for four features the architect only signed off on deprecation for for the technologies no specifications there are no specifications written we just looked at the code anyone can read PHP we just looked at it to find the define the business to find the development scope which is the sum so when developing look at the story then flip over to the PHP and look at the business logic keep in mind the cross-cutting scope alterations both feature and technology clearly modularize it but err on the side of leave it in and leave it broken now this is kind of important leave it broken why would you leave it broken when you are modernizing in place in the nth in enterprise the very definition of that is that you are in you are like in production you are doing you're serving requests and so you need to integrate at the you need to integrate and be able to drop in and that means even returning errors so the erroneous behavior like you see here which is returning a 404 instead of a 500 error on and in the legal input which is horrible API we left it in we had to leave it in because it was a drop-in replacement and we needed to be able to do differential testing between the old and the new so I want to kind of briefly go over a mix it mix in component pattern that we that we use this uses the cake pattern it's focused around this single trait this is a trait that that supplies the functionality around a what's called a URL gatekeeper so this complete this comprises whitelisting blacklisting you know calls out to google safebrowsing for you know just kind of mean people shortening malicious urls and then kind of posting in social media and hiding them right so we of course did our best to do our best to prevent that so but we don't want to be calling google or calling you know calling the the other kind of blacklist providers in test and so we were able to we wanted to kind of have a mockable component like this so here in test we mocked the URL gatekeeper of the URL gatekeeper component and then here in our standard issue play framework rest api test we everything is mocked you see the top line there the private epi short URL controller that's the controller under test everything else is mocked because it's a unit test for that controller and then what you're left with is a simple standard issue call to using the standard issue play a test kind of constructs so how do we do the UI almost at a time so I'm kind of going to breeze through this we trained you I we we got a UI guy we got a couple of you I people brought them on to our team gave them zero training the the funny anecdote here is that they were like you know so the running play with SBT and it's kind of reloading and they're like well how I thought this was like a horribly heavyweight heavyweight framework where hassle you know you have to recompile everything and recompute you know redeployed and everything whether we're like no just you know do your changes and then kind of reload your web browser and they're like oh just like PHP or ruby and so that kind of blew them away the new front end looks like this the front end of alley isn't isn't the most important it represents you know maybe maybe 0 to 1 percent of the the use of the of the system the rest of course is is the is the AP is I'm going to skip over this because I'm out of time but ask me later about patterns for integration integration semantics using a cow who's heard a gearman no one great consider yourselves lucky so what does five thousand lines of PHP look like in in play so I'm going to skip over most of this except for except for this so here we see the boilerplate that we you know that we changed thousands and thousands of thousands of lines of PHP boilerplate to you see our logging constructs are stats decals to get our are monitoring and and our statistics around around performance we see the the parsing of the of the parameters and then here in the for comprehension is just the guts so that's what the five thousand lines turns into those six lines yeah it's all functions in there and those are you know calls into components and of course there's more than six lines but the point is is that the point is is component ization so each of those lines represents a component and a step in the system that are comprised together and is individually testable so why was this a success at the end of the day feature flagging and traffic siphoning because we had a lot of traffic and we could siphon it we could turn it on and off and we could turn it on just for ourselves in production we could turn it off because it or not turn it on for only ten percent of the traffic we always iterated into production clear scoping and estimation of the project measuring everything and of course the the framework that we all love so again we turn this monstrosity I into this saved 30 thousand dollars a year right off the top in AWS costs and untold benefits monetary benefits around maintenance and all those things that I listed at the beginning just kind of worked really really well so I want to end with a funny funny anecdote so one morning my friend of mine who works in security at our organization runs up to my desk he's like oh my god the current DDoS in progress against against Ali it just leaped up tenfold went from you know 10 requests a second to to 100 requests a second what just went on what like what what happened if do you do anything I was like oh yeah when did this happen 10 a.m. oh yeah that's when we flipped the load balancers completely over to the new system the new system was able to absorb a hundred times more traffic than the old system on you know ten percent of the the amount of hardware the ddos spots out there on the internet simply went faster because they were single threaded that we were able to absorb more traffic so it looked like our ddos the current ddos was was increasing in magnitude but we're just able to absorb the traffic faster i think a funny end of that anecdote that isn't true would be like a drop off of the traffic meaning that through our sheer ability to absorb traffic we crashed all those you know little C programs out on the internet but that wasn't the case we just you know blacklisted the IPS and then it all went away then it wasn't an outage or anything it was it was just a it was just a hiccup thanks that's my talk time for questions questions yep it was around the what what I called kind of user stories but what I meant why that was kind of a vertical slice of for the most part was a single API call so at the very beginning it was like the easy ones like the like you know just returned a record there's like a bunch of kind of administrative there was an administrative back-end so the easiest of the easy we would we would modernize and put into put into production so that's that's kind of what I meant by by incremental like taking taking a single slice and then going all the way into into production live to to live traffic and then we would kind of feature flag it on you know when we were you know a little bit jumpy at the beginning we would only feature flag it on during the day so we would turn it off at night back to the old system and we had at hoots we at we have a very kind of to get into this but a very very robust future flagging system we call it dark lunching because it sounds cooler but we were able to kind of get things into production and very very safely other questions yeah excellent question yes part of the one of the strategies for this modernization was to use the same data store and wrap a service around it so we had a we had a we had the service alley data there that that my sequel database on the left there that was the same one that the PHP was accessing as well a previous microservices modernization in our in our Armand Olaf at we actually exceeded a partial database extraction a two way to a database extraction using this amazing tool called tungsten that was able to kind of publish and do like a partial table extraction live and kind of migrated forward to another database in both directions that would in turn be wrapped in a micro service itself but in this case we we were kind of conservative and just wrapped the same database in the future like the future plans of this of course is to write a new service with more robust persistence and because of the micro services architecture will just be able to swap that in as a service without having to like kind of fool around I mean migrate the data and then swap it in as a service implement the same the same interface the question was did we send the same request to both systems and compare the results yeah differential system testing we did do a little bit of that but surprisingly the it wasn't as fruitful as you might expect we did we did a little bit of automated differential system testing but it was a little bit too fragile and didn't provide enough focus in our in our bug testing so what we preferred was to use the existing production system and our our feature flagging to do that but you're absolutely right differential system testing is a super legit way of rolling out a new API a new API in a drop-in replacement yeah we it wasn't very yeah it wasn't very what we tried it wasn't very robust it was we didn't want to write a lot of code it's that did testing itself because of the fact that we had that we had like all this kind of production infrastructure but we could have easily done that we didn't perceive the kind of exact results even though it was important we didn't perceive the exact response codes and and payloads of our API calls as the crux of the of the system the crux of the system was the deployment the drop in the kind of and the riorca tech chure and stuff that we didn't we didn't want to focus on that when we had running code that calls in already and kind of we could check the error codes and the kind of the interpretation of that live but in a system the drop in system where that kind of those contracts were way way more important I think a differential system framework differential system testing framework would be way mom would be way more hard other questions I think I'm at a time thanks a lot you