scala.bythebay.io: Jamie Allen, The Future of Services
Recording: scala.bythebay.io: Jamie Allen, The Future of Services
foreign hi everybody so uh most of you probably if you've heard of me at all uh have heard of me from when I was at light Bend I spent four and a half years there and I left there in mid-september to join Starbucks and for me this is a really big change because I've been working in open source for four and a half years there you know working in Consulting for so long that this is the first time I've worked for a large corporation as an employee in my entire career and I'm old so this is for me a big step like this is a complete change from the sort of things I've been doing before and one of the great things about it is that uh you know for for those of you who didn't know me I wrote effective akka I'm co-authoring the reactive design patterns with Roland he's doing most of the work so now I say I'm a contributor but when you work for a vendor you don't really have a lot of experience with your Technologies and that probably sounds bizarre but it's true you join a vendor and then all of a sudden you are out there showing people how to use stuff that you've never used before yourself like Technology's come out from that vendor and you have no idea how to operationalize it it's pretty funny so the the reality for me was that only through building software only through building stuff can you get any idea how things actually work I know that's probably a surprise but you know for me this is what I wanted to do if I was going to lead light bend it wasn't going to be to go and do another open source company or something like that but go back and build things so I had a really good idea of who all in the community was using light Bend Technologies and what they were doing and what we were doing at Starbucks seemed like really a great opportunity for me to get involved in because it's totally Greenfield and I could totally mess it up but at least it would be not you know bettered by the sort of things that already existed there so we'll see how well I do so microservices are a really funny thing because they don't really exist in a meaningful way if you look all the way back to when we were doing SOA the problem was that we were really trying to build services in a very coupled and and not well isolated way and that led to the failure of the entire movement now when you hear somebody talk about doing things with soas and esbs and stuff like that it's usually on failed implementations right but now we know that we all want to move in this direction called microservices and nobody's telling us how to do that and that's really frustrating for anybody who's got this mandate from you know their their business they they've got words like Cloud they've got words like microservice they've got words that you know they hear from their peers and they want to be just like them but how do you build them nobody knows so you got to kind of figure out what the best practices are from all the people out there who have been trying to do it and this is led by people at organizations like Twitter at organizations like Netflix the ones who could take a chance on their architectures because they weren't encumbered by you know the implementation of cobalt things that have been written 40 years ago what are we trying to achieve we need accelerated teams they can't be sitting there working in you know depending on everybody else for all the things that they have to do we need faster deployments we want to increase throughput lower latency we also interestingly want to hopefully use less machines and that kind of flies in the face of this this unofficial rule that we all know that if we want low latency and high throughput you have to use more machines right that's the idea and yet the real problem is that we're using things so efficiently and then we also want fewer dependency nightmares so microservices Size Doesn't Matter the number of services doesn't matter buzzword compliance doesn't matter every organization I've run into when I was with lightbend was doing in a lot of different ways and some of them had as many as 1700 Services running on six-figure number of virtual machines I mean that's just huge numbers of infrastructure and and issues dealing with them think about trying to run out new versions of stuff and how are you going to manage the dependencies between all of those different implementations that's really difficult to do so what we want is isolation and for those of you who have seen me talk about this before this part really hasn't changed we know that we need to have it at the API we can't be calling through clients if we're calling through a client API then we're going to be stuck versioning the the implementation of the thing using the client at the same time as what it's consuming so now you're coupling your service versions that's a bad thing right this is why you see people talking about using restful apis over you know http um we need in the source code we can't be sitting there dealing with guava upgrades across our entire code base Ben Christensen talked about this whenever he gave his talk on microservices.com but the last one is data it's very easy to get sucked into the idea that I'm just going to have one database behind my all of my services and then if the scheme ever has to change you've got to figure out all the different places that are worried about that you know that that schema now being you know representing the data differently and that can be extremely painful how easy is it to figure out all the different places that are accessing this one part of the schema there's one table there's you got to search through all your entire code base there's nothing pointing you there so we don't want to avoid that Services have to own their data and this is the Titanic right and the problem here was that the bulkheads didn't go all the way to the top there was no isolation the ship probably would have been fine if the bulkheads had gone to the top and water couldn't spill over every bulkhead as it started tipping forward right as it started listing because of the water that was flowing into the front of the ship so lots of people died hopefully our software doesn't have that problem but we have to accept that this whole monolith the consistency idea isn't going to exist in any kind of Highly available application there's just no way that's going to happen if you're going to have all these services and all these databases we know that they're going to be running on different boxes unless you're going to sit there and put it on some one big AWS thing which isn't going to be highly available right that's not going to be working so we've got to have the understanding that data is never going to be fully consistent at Starbucks people say to me how many employees do you have and I'll be honest I've asked some people myself and we don't really know that's probably something I shouldn't be saying don't tell them Andy but I mean really if you think about just the number of people working at various stores we think it's a two three hundred thousand we don't know exactly how many employees are there right this second because of you know people coming and going they've got you know uh maybe they go off to college or something like that maybe they found a new position maybe they're joining us there's just so much change happening all the time and then whenever you talk about how we have to build systems around the company and you say that there's going to be eventual consistency and people say no no we can't do that we don't even know how many employees we have but we can't you know be inconsistent about you know how many stars you have whenever you're accruing um you know stars and loyalty for the coffees you're purchasing we got to know exactly that amount and really to be highly available we have to accept that that's not true we're going to give you our best guess we want to give you an idea of how many stars you have but what if we can't at any given moment it's best to give you our best guess rather than not respond at all that may mean that you think oh hey I have more stars than I have because we haven't we haven't resolved the fact that you just spent a couple of stars to get a free coffee well okay hopefully we have some fraud procedures in place to make sure that that doesn't affect us negatively but we're also we're selling coffee by coffee here we're not selling this massive product for thousands of dollars right and distributed locking and things like cross service consistency will fail if you even try to attempt it anybody give this a try at all has it worked for anybody now and yet I hear people talk about this with various Technologies I'm going to use a distributed lock and then I find out that you know that totally fell over on them because at some point in time machines weren't able to communicate and somebody was holding on to a lock and couldn't move forward and when you get that sort of thing and you try and make that work this is my favorite picture has anybody ever seen this before the entire system you know gets up and running looks like things are going to work absolutely beautifully and then it gets this one point and everything completely Falls over did anybody ever see these Robin Reliance before they're hilarious cars there's a great uh top speed about this where he was literally driving around Sheffield in the UK and tumbling the car over left and right so when we have a monolith we're thinking in terms of consistent data we're thinking in terms of synchronous interactions because everything is located within a single virtual machine we're thinking about releasing things together and scaling and mass but we know that with microservices we're going to have this eventual consistency across service boundaries because we cannot guarantee the transactions will work across service boundaries we're going to have asynchronous interactions we're going to have releases that are independent and scaled where needed right scale only the services that have to scale independently now this is the thing where I'm going to sit on this slide and start telling a few stories I've worked at a lot of different customers for a long time who are doing stuff with microservices trying to use reactive Technologies functional programming all kinds of really cool stuff and the real problem at some of them was trying to figure out how they were going to determine what a microservice would be has anybody run through this where you you say you're going to use a microservice architecture well we know for example at Starbucks that we have members in our loyalty you know program we've got a membership right there are people who are part of the Loyalty program but then there are things and characteristics about when they expire you know when they uh they may no longer be a part of a loyalty program they're the programs themselves because in Japan we've got a different program than we do in the U.S you earn points at a different rate there's all these services that you can almost sort of see but you don't understand what they are until you start understanding interactions so at lightband you know we we when we were thinking about creating the log on we were thinking in terms of how we could represent bounded contexts but we never told you how you are going to identify your bounded context and that was really up to you as somebody as an organization that has to understand your business and that's tough to do if you're sitting down and just saying that you're going to do domain driven design how far are you going to take that are you really going to break things down by every single bounded context where do you identify them how do you take people who are entering user stories in jira and turn that into actual bounded contexts I don't know and so what if you take something like event storming has anybody done this event storming is where you're sitting down and talking about the commands the triggers that influence your system and the events that result from it so now you start thinking in terms of interactions instead of just these things that you know exist in isolation and you know what's really cool about that now you're also starting to think about composition that's neat because before I couldn't see where the composition was and I could start seeing it when I understand the events that result from a command how does that flow into another command how does that flow into another result and once I start seeing that I can start figuring out my Aggregates Chris Richardson is running around talking a lot right now about doing this using event storming as well and he talks about binding your Aggregates and I'm gonna have a little bit more to what he has to say in the next slide but this is really about feeding your teams if you're in an organization where people are just typing stuff into jira and your team is supposed to figure out how to turn that into a microservice architecture I guarantee you're doing it wrong you have to be doing more having these interactive sessions where your product people and your engineers are sitting down and talking about how the inputs come in and the outputs go out will result in your ability to find your aggregates so what are these Aggregates right their data they must live together for some reason right aggregations of your domain aggregations within your domain is a way to think of it um and this is really where you have to think about what data has to be consistently updated atomically updated together therefore if you know that you can't have transactions across a service boundary well this is where you're going to find it you're going to have that data located within a single service or maybe that data is frequently accessed together and it doesn't make sense to split it apart for latency reasons you know that you're going to be calling out to this other thing every time somebody is looking at this one value so why not co-locate them or there's no way to identify a primary key relationship between them everything should be based on identity if you have Services calling into other services and it's not based on a primary key identity relationship then you're making other services figure out how they're going to allow you to consume that information and that's almost like you're now coupling your services in a way that shouldn't matter maybe those things belong together so that's usually the things we think of first is primary key relationship and consistency but I would say that scaling is also a factor there as well right because I don't want to have these barriers these artificial you know lines in my system that just exist because I say this has to live separately when really it doesn't it's just me saying that there's a primary key relationship that's not the only reason to think about splitting stuff up and then you have to think about all of the things you have to deal with for resilience if resilience is your number one goal and at Starbucks it's one it's one of our highest goals because we want to be a great brand there are a lot of great Brands out there Kleenex is a great Brand Apple is a well thought of brand Tesla is a pretty well thought of brand right everybody has warm and fuzzy feelings about them people at Starbucks want that same thing as well and in order to do that with the level of people who are using our Mobile Pay which is extremely high we need to make sure that people don't have a bad experience doing so that they feel comfortable you know paying through their phone and that means we have to do a whole bunch of different stuff to make sure that we have up time that includes being asynchronous right having back pressure having the bulk heading between our services that isolation to make sure that what happens wrong in one service does not impact another circuit breakers which will help you deal with the Thundering Herd right back pressure is great to have at certain levels but if you're not going all the way back through TCP if you're using some other protocol at the network level you might not have the ability to propagate back pressure at some point you have to say that I'm not going to be overwhelmed anybody remember the Dos attacks that were happening when all the internet of things went crazy a couple weeks ago that impacted us we weren't happy about it right we don't want that to happen so we need to make sure that we shut off the Thundering Herd coming at us at some point and circuit breakers can be kind of weird because this is where you're arbitrarily saying that for 250 milliseconds I'm not going to accept new traffic right it's it's almost always this kind of hard number but it's not based on anything more than some guess of what's acceptable all right after 250 milliseconds they'll try nope can't do it now gotta do another 250 milliseconds right you could do finer grained but that's harder means your computer is doing more work you can do it less Court more coarse grain but that's probably even worse um we want clustering where necessary if we can avoid it we will we don't want the extra complexity of trying to figure out whether or not we have to keep a cluster of nodes aware of each other if we can instead think in terms of you know individual nodes who grab something out of something like Kafka and run it through a series of Transformations and put it somewhere else without any knowledge of any other server doing anything else that's a simpler architecture because I don't have to Cluster and if something went wrong I can just retry pulling that thing out of Kafka because I never acknowledged it it's a very simple form of resilience and you can only do it if you have three things one you have commutative data order doesn't matter things come in and it didn't matter that something that actually happened after it came in before it okay so you're commutative if you are associative and you're not worried about any other events happening to be able to process this one event and if you're stateless right stateless is very important as well and then we want parallelism which is almost exactly what I'm talking about whenever I say that I don't want clustering if I can be fully parallelized and not pay the costs of amdahl's law then I'm in much better shape at Starbucks we're going to be pulling in Colton Andrews and his new company called Gremlin these are the people at Netflix who implemented the semian army so they're doing pre-betas right now I'm giving them a shout out because I'm a big believer in that take ourselves down in production knock our services out on ourselves find out if we can withstand it because if you're not going to that level of rigor when something does go wrong you have no idea what to do as a result of it so we may as well knock ourselves out right and yeah it's going to be totally non-deterministic I can't say that what knocks you know if one server goes down you know at the time it was doing certain things that I can replicate that behavior in the test again I can't necessarily do that another completely random you know knockout of a server or knock out of an availability zone or whatever you're using you know will have different results on your system can you handle all of that it's much better to find that out than to find out in a completely non-you know supported way where you're watching it and understanding this and then there's also supervision which is a really big thing to me I need to know that somebody's watching every process out there and and if I can every thread that's hard to do but at least then I can build in that whole idea of a domain of failure I can understand not just that a customer was able to create a you know a membership in our loyalty program one of the things that can go wrong in my domain we're really good at building types for all the things that are good we're really bad at you know coming up with types for things that go wrong they can be types in our system as well they can be messages and then we can react to them in the same way we do good things so we want immutable deployments is anybody doing immutable deployments here this is where you're fingerprinting uh fingerprinting your your artifacts and saying that I know exactly what binaries are in here along with the configuration associated with it so I know if anybody went in and tried to change it well it would be another fingerprint and I would see that I'm not running consistently the exact same thing on every box whenever I'm releasing that is a warning to you that you don't have a consistent deployment we need to know exactly what's running in our production environment the flip side of doing all these really cool things is operational complexity it's not free and this is a hard thing for me because I've got to get something in production and I only have so much time to do it it's great to say that we you know want to be agile and everything like that but every business out there has people whose necks are on the line to get something out there in production by a certain date for various reasons and that means I have to think about a whole bunch of stuff that my developers have to do that has nothing to do with actually coding in some cases Swagger API definitions for anything at least external right I don't care that we swaggerize every API in my system because to be honest with you if they're not being seen by somebody outside of a system then I don't care maybe someday we'll get to that level of rigor but right now we gotta we gotta put stuff in prod orchestration how am I going to make sure that whenever I deploy something out there and I want to have three instances of the service out there that three are running right I need to make sure that all three are up if a node goes down or jbm goes down or what have you bring it up again or bring it up over here right I've got to think in terms of service Discovery if I'm going to run in the cloud I've got to worry about making sure that I know where my things are running because I've got to still route stuff to them and that's where you see proxy and you can't proxy if you don't know where stuff lives and it has to be dynamic you never know where that's going to change because of nodes going down and service gateways which are going to make sure that stuff are routed to the appropriate places stateless aggregation failure management versioning has anybody done versioning and microservices were you the same microservice multiple versions running at the same time yeah I know yeah so when I I worked at a place in Philadelphia for a little while we were using osgi and uh we had some struggles around this ourselves one of the promises of osgi was that you'd be able to run multiple versions but we never really quite got there and this is what eight years ago or something like that so you know it sounds like a Panacea but if you get there and you do it right then you have the ability to deploy a new version of a when you can rev B but C doesn't need to be revved at the same time right because you don't care you have to think about latency and how you're going to track it but then you also have to do tracing and monitoring this is one of the hardest things there are very few tools out there that will help you understand in a distributed environment a flow this is where Zipkin is a really big deal here at Twitter right if you can find tooling that allows you to understand what happened here then there then there so you could see a chain of events that took place across Services you're in great shape but you know what it becomes a big data problem now you've got so much information being tracked across all these servers that you've got to find a way to meaningfully parse it and pull it out and figure out what it's doing and suddenly you're running spark just to understand your logs yeah yeah you are I mean seriously so you have to do sampling you can't just say you're going to get every single thing that happened on every single service because it you'll overwhelm yourselves with data at light Ben one of the first things that we ran into whenever we released monitoring for akka was that it was a big data problem and if you monitored 6 million actors on a single machine with all the events for whenever something was enqueued into a mailbox to when it was dequeued for processing to when it was done processing you end up with a lot of data and we melted our front end we had this beautiful front end you know it was gorgeous it looked fantastic but it absolutely couldn't handle all the data that we were trying to put into it to show you what was happening so at some point non-deterministically your data is going to overwhelm your systems and you have to think about what you're going to do when that starts happening how you're going to slow that down because you don't want to lose things you still want to get your alerts but you need to make sure that you're not taking your system down just with network traffic on aggregated logs you will so run books is anybody else here doing run books this is something I learned from Katie McCaffrey actually here at Twitter she uh is operationalizing a whole bunch of services here and she gave a talk at uh at uh matadorama 2016. and if you go to this GitHub page you will find her template for run books I've added a few things to it I haven't been able to share it yet but this is the idea that if I'm going to have alerts out there for all my services they're going to be talking and letting me know when things are going wrong well you're gonna have people who are on call and every single one of my development teams even though we're not in production yet has an on-call person every Sprint just so they're used to the idea that somebody's got to be signed up for this and since we don't know exactly what date we're going to go into production it's pretty much random anyway you may as well just sign up and say it's me right now right um but this way when they get an alert they don't have to figure out what does this mean and what do I do as a result of it and this is also helping you with your organizational you know knowledge in case people go to other teams or leave and decide to work or not work they go and live their lives on a beach somewhere because you know they could people retire that happens so when they leave you lose a lot of institutional knowledge these run books help protect you against that so the benefits scale applications independently scale teams independently ability to be responsive hey that's all good stuff that's it really questions Jason right so the question was what format do we store our run books run books I mean right now we're doing it in the same format that Katie does which is a markdown file right they've got to be aggregated in a production place we actually keep them right now with our production deployment artifacts that's the idea but because we aren't actually deploying anything to production we can't say that we are using Concourse CI which is a spring product a a pivotal product and the great thing about this is it allows us to Define governance in our CI CD prod flow so now we can say do we have all of the artifacts we require to go into production before we push this thing out that's our idea and run books will have to be one of the artifact artifacts enumerated in that list so yes you mentioned versioning and of different microservices so what kind of have you created like dashboards or what if like The View that like maybe you as a product owner or whatever like a monitoring like you're using something or have you contributed okay so the question is for dashboards for versioning and stuff like that no I haven't at Starbucks yet because we're not yet in production with this stuff right what I have seen people do is use the standard like rafana type things where they just push out a view for every single version the really hard thing is how do you make it look sensible because you end up with this here and that there and this thing over here is a window that you can click through but you know I haven't figured out a way to say how do I localize them so I know the two versions are right next to each other so that will be something I want to do but right now no yeah any questions all right well thank you [Applause]