jamie allen revised 3 17 16

so I'm Jamie Allen I'm the Senior Director of Global Services for light Bend i'm going to say typesafe a lot tonight because i still haven't been able to convert my brain and if you're wondering why we're now light Bend I just want to really quickly explain that one thing about type safety is it's something that programmers value we believe in it we wanted especially scholar developers the problem is whenever we're out there talking to executives the people with money who buy things they have no idea what type safety is and they don't care they think we sell fonts they think we're trying to help them type their emails better and that's not good for business and another thing about is we don't know what the future holds for technology if we'd renamed ourselves with something like reactive well then who knows what the future holds and we don't want to bound what we do based upon what is going on right now so you know pivotal did this as well they took springsource you know whenever they were acquired by VMware whenever it was fun out and they became pivotal they choose a great name that doesn't found what they do as a business and that's important they're not the cloud foundry company they're not the spring company they're a company that solves problems we have the same vision so hence light Bend we think this even though this wasn't the reason it was chosen I feel it means a gravitational pull toward our philosophies the things we believe in when we're trying to build systems and this talk is going to be largely about what we believe people should do when they're building systems and how we're trying to enable that so that people aren't making the wrong decisions so first of all I'm going to talk about what I think microservices are because I think they're very poorly defined and then secondly I want to talk about what do we want out of our service architectures right after that I'm going to introduce log on this is a Swedish Dessay correctly thank you we have Swedes this is a team from speedster by the way did I say that correctly speed meant I'm sorry and they're from Stockholm I think and they're there you know Swedish so if I said if I'm saying it wrong they'll let me know okay okay so i got gotten berg gotten Bergen southern Sweden so there could be an accent then all right and then after that we'll take some questions now I'm going to start off with something I really hate doing I'm in a quote gartner group right but they have a very valid point and that's the idea that we're building systems all wrong for the way we have to deal with data in the present and the future and this largely has to do with all of the different channels that we're getting data from so how are we going to sit there and deal with all this new traffic and new channels with our traditional architectures which are already overwhelmed and you know what are we trying to achieve when we build these systems we want accelerating our development cycles so that people aren't sitting around waiting for other versions of stuff to be released that we can release our stuff we want to reduce the dependency nightmares between our individual services and then we also want to increase our application throughput and by increasing the application throughput I'm not necessarily talking about low latency I'm talking about dealing with failure I'm talking about being elastic whenever we have all this kind of traffic coming at us and being able to scale independent services so that we don't have to throw a whole model if out there just so that our inventory service could scale right micra services is a really bad term it says nothing about what they're supposed to do now when you're a company that's a great idea right I just explained that but whenever you're trying to describe an approach to building systems this is horrible size doesn't matter it's not about trying to create little services it's about trying to find a way to separate concerns and then making sure they don't step all over each other so when I talk about services and I tweeted this and got into a you know really nice discussion with some people I want isolation that's all I care about the things that I separate out from my services so that nobody else has to think about them and they can ask me for them I want to do this Ben Christensen had an amazing talk at microservices calm i think it was i highly recommend you see this if you haven't been Christensen formerly of netflix now at facebook and he talked all about the nightmares they had when they had a common dependency between multiple services so think in terms of guava if I have anything in my you know my code that represents a util library or a miscellaneous library or core something shared between services and there's a dependency in there on a third party library and all the dependencies they pull in you are already in the spaghetti code nightmare it's impossible to upgrade anybody have this problem no right of course not we all do and this means that we have to take very extreme approaches to isolation that means that we can share code within a service we cannot share code across services if we been made the point if we put a client out there that other people call to us through we are already bounding them to our own dependencies and that's coupling we know coupling is bad but we still do it so we shouldn't we want isolation at the API at the source code but also at the data Chris talked about this I'm going to be saying a lot of things that Chris already talked about which is great because I can blow through them he already explained them but in our data we want to make sure that our data is completely separated we only want one view of what an address is we wanted to serve all of the other places that need to know what an address is for a customer so a customer may retrieve from the address service information about the address and serve it to you in a specific form that you're asking for for your service that way you're not dealing with the complexity the service is dealing with all the complexity inside of itself and this is where we get the bulk heading that is so important this is a picture of the Titanic from you know the Wikipedia and the real problem that the Titanic had was it didn't have isolation between its bulkheads water could spill over a compartment bulkhead and that meant a ship started to tilt downward which meant war waters spilled over other bulkheads until the ship tilted up and down right and broke and all that other stuff that was in the movie but the point is this is not affected bulk heading we have to have that isolation between our services we want resilience and also independence and isolation this will give you the ability or teams to work completely independently of one another right and not have to care about I have to rev a new version or something like that other people don't care so long as you're still exposing what they need they're fine it's not their problem and now with some additional operational complexity you can be nimble right that's what we want to get to we want realistic data management I want to tell a story about this and this is going to tell the story of a micro service before all of this micro service you know terminology came around but it's a vision of how a large company had to split off functionality from a monolith a monolithic datastore and a monolithic application I was working at a large cable company they have 40 million customers out there and all they do is sit there and say you know you've got to count you've got you know devices that you can watch your stuff on and then you've also got you know what you're allowed to watch and if you say I want to watch Game of Thrones I want to call up I want to subscribe to Game of Thrones on HBO and be able to watch that you call in they update their information and some big monolithic relational database store that's what they do and that worked fine for years it was great until on-demand came along and on-demand suddenly changed everything not suddenly actually was more of a you know a time-based thing at first there were just a couple things you could watch on demand and that wasn't such a big deal they had this relational data store they could do the rights it could do the reeds and the DBA sit there and work really hard trying to make sure it do both really well but in the end they couldn't make a way to show entitlements without a complex join and so they were struggling as on-demand became more and more prevalent more content was available more channels were provided suddenly people were watching more and more on demand and this was killing their ability to serve people at their highest level needs seven p.m. on a friday that's when everybody wants to watch on demand can't do it sorry we're out that's a bad thing for your business right nobody wants to stick with the cable company that does that that's why they go to companies like Netflix who incidentally use a very microsoft microservice oriented architecture so in 2011 we were sitting around looking at this model let them we said you know what we should do we should split the entitlements off and that means in essence we're taking what was a monolithic datastore because it really wasn't the application that was the problem here we could scale more instances of the j ee application the datastore was our problem consistency was our problem and so instead we say let's make that that relational store a right based store whenever somebody calls up and says I want to watch HBO it's going to write that information into that store and then on the read side we're going to cash that information in a completely different data store ok now we joke about you know the three hardest two hardest problems in computer science are naming things and caching and then off by one errors right but the point is you can do this and actually have a sane architecture what we did was first of all we thought if that store could just emit an event we get that event we put it into some some you know message queue like rabbit and then we consume it on multiple you know handlers on the back end it all subscribe to the topic they're just grabbing stuff off and calculating entitlements on the fly and then putting it in that store the problem becomes time you don't know that any of those events are necessarily going to happen in that order or that they're not going to get dropped or heck rabbit might drop them rabbits not infallible what happens so what are you going to do if that's the case if you're lucky enough to know that you cash is out of sync well then you can rebuild it from scratch but that costs time and downtime that's never fun for a business instead what if you just constantly recalculate people's entitlements and you figure out what your your bucket size is for accounts that you want to handle you minimize the latency between how often you recalculate am I calculating a hundred thousand people or I calculating a thousand per bucket right and then I'm just constantly always recalculating and in doing so I don't care if I ever dropped an event in that sequence of events doesn't matter so long as the big monolithic right store received the event the read store will find out about it eventually and even better the read store we use something called rioc which is a dynamo store which is much you know key value store basin you can have a ring of them and it's going to do all the things that Cassandra does but give you key value semantics awesome well the great thing about this it also gives you sessions and now one of the hardest problems you have in the scenario is how am I going to delete out accounts when people call and say I don't want to be a customer this cable company anymore one of the hardest problems you're ever going to deal with instead you just let the sessions wash it out if I don't update that customer within a day they go away they get tombstone the database handles it in its own way not my problem not in my code easier to deal with so this was us creating an entitlement service which had an API which provide you either a protocol buffer view or a JSON based view of your entitlements by you know account by device by both account and device and that way we had these buckets and yes that meant duplicated data and duplicated from what was in the right store that relational store from before but who cares that's not my problem the fact that I get consistency over time and I'm never worried that I will never see this event is how I have a self-healing resilient system so getting away from the idea of transactions is a really big deal if we had SAT there and said what we need to write to the Oracle sore and we oops if we wrote to the relational data store and we wrote to rioc right as a single transaction well then we're going to have all these semantics that make it extremely hard for us to figure out did we write to both and what do we do if we didn't get one of them pack with that instead of thinking in terms of this idea that I've got a distributed transaction I allow for the idea that I can compensate for it over time that's how I can be resilient and now you can also build systems with domain specific behavior for when things aren't going your way instead of trying to wrap it in some at transactional thing which doesn't work it's a blunt tool right especially in the distributed context so let's think in terms of getting rid of transactions instead think in terms of compensation instead of this idea prevention Kevin Weber is one of my co-workers at light Bend and I think it's a really great quote consistency is also an anti-pattern I work with a lot of vendors out there and one of them says to me consistency is what we're known for but you know what we really need to be able to scale and be available all the time so we need to be able to put it out on multiple servers and be consistent all the time and I look at them and I say that's really hard and that's because my sales team will not allow me to say things like that's impossible because of physics right it's true you cannot have consistency and availability you can have a sliding scale and you can make choices about how much consistency you want but you do not want to sit there and say that across multiple machines I have an absolutely consistent view at all times this is where we get into problems with things like data fabrics right they sound great the idea that we're just going to have distributed locks and then we're going to be able to put stuff out in this data fabric and share data across all our services no no that's not ownership and then you also get into the problem of how you're going to you know version one service versus another service don't do it think in terms of ownership isolation not shared so Chris did a really great job of talking about seek urs and events sourcing I want to talk about one more distinction there's the idea of a command there's the idea of an event commands are what come in they're the things that tell you what you what just happened like the person calling and saying I want to watch Game of Thrones the command goes into the Oracle store the event stores the effect the things you did as a result of that right so that's the separation we have between command sourcing and it and sourcing command sourcing is a really great thing to use if you ever want to be able to replay failures so in systems where failure is a really big deal and replaying them and something you want command sourcing is a sort of thing you need to think about events sourcing on the other hand is how you rebuild state because you don't know that you're going to get the same events if you replay a command at a different time right you don't know that so think in terms of rebuilding state in terms of what you did as a result of the command as opposed to the command itself we want asynchronous api's and right now the the thing that we all really want to do is restful services but rest is a synchronous interaction request response we can do it using asynchronous calls using things like futures and we say that's fantastic except you know while we may be doing it in a non-blocking fashion so the the threat isn't wasted we still have these ideas of I'm waiting for a response what if you didn't have to go that route yes at certain gateways you front things with the RESTful API but behind that when we ask for customer information we say all right give me the customer the customer then goes and aggregates data from multiple other services behind that restful api and those can all be asynchronous and streaming wouldn't that be great that's how we want to build systems in the future that's the way light Bend has been thinking about microservices going forward and the reason i have a postal you know stamp here is is the only metaphor i could come up with for the idea you send something and you walk away kind of thing right somebody may respond to you in some time but you're not sitting there and worrying about it you may have some semantics building to your system around this stuff but don't build them into purely synchronous request-response interactions now we also want immutable deployments how many people have immutable deployments in there you know existing deployment infrastructure a couple this is getting to be a really big deal especially with containerization the idea that i can wrap something up and know exactly what configuration i tied in with this deployment footprint this jar this war or whatever at the same time I know I've got a I've got a thumbprint or something it tells me it's exactly this one and nobody can go in and change it right nobody can go in like Wally here and mess around with the individual configuration settings in production and mess with it so you don't know what's on each box that's going to be the future being able to deploy these services in an immutable way we also want to expose only the tip of the iceberg we want to mask all the complexity of how the fact that I had to get the address from another service what's happening behind the scenes the caller asking for customer information doesn't know or care I'm just getting the information and bring it back to you you don't care about all the ugliness below the water we want ddd but it's not a requirement one thing about that example I told you about already with the cable company was I never once said bounded context I never once said aggregate route those are all you know great terms for helping you decompose greenfield problems they can be trickier to think about in a real world where you already have a legacy system where you're trying to think in terms of what is my pain where's the thing I need to fix and deal with right now it may not map directly to a DDD approach and more importantly a lot of people in your team may not have read these books anyway so who cares if you call them bounded contacts aggregate routes that's great whenever you're having a discussion about using ddd but if you're not just think in terms of what your pain point is now you're splitting things out and how you're going to isolate things and in essence you're probably doing it anyway if you want to read the book that's the one that Chris was meaning talking about domain driven design by eric evans another one that's much more recent it's implementing domain driven design by a fellow named bond burnin Vaughn is an independent consultant out in Colorado and also a really big Scala and akka advocate here's a new book out that's even better called reactive messaging patterns using the actor model and it's a real mouthful but it's an excellent book and it also ties in his ddd concepts with acha so that's a it's a great book I highly recommend I should have put it on there and we want this idea of acid be to the old acid simply doesn't scale the idea of atomic consistency isolate but not the isolating I'm talking about here you know that that's not the way we endurable I forgot the D in this case we want to be associative and commutative and by that I just mean I don't care about order and I don't care about how I group things the more I can build systems in this way in mind you it doesn't map to every problem just like here I could say that if i add 3 plus 2 it's the same it's 2 plus 3 which is the same as 5 right that's great grouping and how i did it the order of it didn't matter but then you throw division in there and suddenly it matters an awful lot 6 divided by 2 is not the same as 2/6 right but if we can keep our systems in terms of associative we're grouping doesn't matter and by that i mean i don't have dependencies on things where i have to pull something in a given time or i don't have commutativity alright i do have commutativity where order doesn't matter then I have a linearly scalable system as an example I really well-known you know consumer electronics manufacturer and cupertino they have you know devices and those devices sometimes things go wrong on them right and that means that you're asked hey do you want to do something about this and you say yes or no if you said yes that something gets sent back to this company and then it goes into Kafka and then they've got to you know run it through some transformations and put it into Cassandra awesome they said well hey you know can we do this using your platform we said yeah you know why not who cares that's easy but disorder matter no doesn't matter what order these reports come in is there any dependency on any others no well then you're having this great solution where you don't even have to worry about things like clustering because you can just scale up nodes based upon how much traffic is coming through linearly scalable where every node you add your handling that much more traffic awesome simple because we don't want complexity right and an idempotent by that I'm talking about the idea that if I repeat something multiple times I get the same result that can be tricky in the face of side effects side effects are always exemplified by the idea of logging right whenever I log the time stamp is different okay yeah that was a side effect in everybody you know kind of rolls their eyes but think of it in terms of everything else you're doing if there's even one thing that's slightly different you're not idempotent right and then distributed being able to have multiple nodes up and running doing this work so there's no one single point of failure this is how we scale systems this is how we are elastic this is how we are resilient so we will have additional operational complexity for our DevOps teams our operations teams our people who are making sure stuff are running even if that's us and the reason for that is we're going to be pushing more individual services that are out there with their own databases running completely independent of each other that does mean a little more work for these people they should be happy they'll have jobs will have jobs somebody's got to make sure this stuff runs but the point is by doing so we've scaled our teams we've scaled our code we scaled our data and now we don't have dependencies we have isolation it does come with some costs it's not free but that isn't the worst cost to pay the worst cost to pay is how are you're going to refactor guava out of 500 services whenever you want to rev one version that's going to be your real pain so what is low gome you know when we came out with Logan there was a really big debate about what we should call it and of course you know it has the word lag in it if you're you know from in English background and that doesn't sound particularly good for any kind of service you want to deploy but we didn't care we said you know what we really liked the name because of what is spouses in Swedish the idea of something being just right right just the right amount we're not saying that this is going to be small or big or anything other than it's going to be just right and so we went with it you know and people laughed a little bit when it came out but you know most people kind of gotten over now and then we came out with a Java API and not Scala and there's a couple of reasons for that one of the more fundamental reasons is the idea of what we went through when we we purchase play to write that wasn't something we built and then we purchase spray and in both cases we had to take them and make them have a Java API and that was hard it wasn't something you could just bolt something in front of as a result the experience for java developers was substandard at this point we finally feel play to from a Java API has equivalent semantics to the Scala API finally but it's been four years and we're not a big company we don't have tons of people to throw at this problem you know that's why you sit there and say well play has global state that needs to be you know taking out so you can't have to play servers running in the same class loader as if you should write nonetheless these are things that we want to fix they aren't the highest priority that's how things go with Logan we had a couple of choices to make we could put a scala API on front of it and then try and tailor for Java later that hasn't worked so well for us so why not start with Java and bolts collar on later but we have a couple other problems as a company this is nothing to do with our valuing Scala a lot of our customers come to us and say we really want to use your technologies we want to use the Java API but there's nothing for us there's no conference to go to there's no meetup groups talking about how to use akka from the Java API right nobody does this it's not cool well we need to do that for these people when fifty percent of your customer base is out there using akka from the Java API as painful as it is think about every message that you send one line of code and Scala is 50 to 100 lines of code in Java when I teach fast track to akka with skylights two days when I teach it with Java it's three just because of the typing I'm not making this up it's hard that said there's a lot of people doing it right and these are also people who are more traditional in their approaches and when you talk to scala developers they are likely to adopt things quicker adopt new ideas like the approaches chris is talking about well the java side not so much they're getting pulled in whether they like it or not by spark the people who said I'm not going to do Scala they're doing Scala now because they know there's a fundamental difference in the experience when you're using spark from Scala as opposed to Java especially whenever you're doing anything through a rebel session right you can't do that from Java all you can do is have batch jobs so this is starting to shift but in the meantime we wanted to put this out there a to let job of people know we care yeah it's true B because they need the most help right they need an experienced it helps them build these kind of systems and then see it does help us because it's harder to do the Scala one first and then do Java later so you know a little give and take their and love them has four components to it one is the idea of a service API I'm going to explain each one of these in more detail shortly a persistence API a development environment and a production environment so first let's talk about the service API you can write synchronous restful api if you want that's fine but we're not going to let you do it easily we're going to push you toward asynchronous streaming API s because we think that's the future we think that's the way people should build their systems and we want it to be a first-class concept inside of our framework as a result we think there's a lot higher resilience and you know elasticity that comes from not blocking on threads and so we're always going to be pushing that inside of our code and hopefully yours as well and a service itself this is where you get the bounded context in ddd that doesn't mean that what you build has to conform to eric evans or anybody else's idea of what a bounded context is like i said solve your pain we have the persistence api and we think that the crud store the create read update delete is an anti-pattern we believe that the update and delete are the problem we think creating is fine we think reading is fine but not the ud and therefore we are pushing people into the sea q RS model and we have Cassandra already built in to log on so that you can use it as a Sikh urs right store and a reed store with different key when it called key the spaces key spaces thank you and that gives you none of this baloney how many people here to work with things like toplink hibernate yeah a few hands go up toppling kind of dates me huh Wow but you know the idea that we're going to use these object relational mapping xand boy way back in the you know 15 20 years ago I suffered through a lot of arguments between the poor database administrators who had to sit there and try and figure out how they were going to optimize on the right side and the read side for the system and how they were going to organize their data versus what we need to do to organize our domain in our code in Java and JE and even spring later on it's a really hard problem to solve and inevitably you end up sitting there and hibernate rewriting queries and doing all these sessions with your you know your database experts trying to figure out how you can write something that's both performance and not overly complex and it's really hard to do so getting rid of the idea of an object-relational impedance mismatch is a big deal but then you can always determine your current state Chris talked about how you can have snapshots so that you know every midnight or something like that you roll up the state that's been accumulated over all this time and now you know you don't have to recalculate everything this is actually formalized a lot through things like the lambda pattern that Nathan Mars came out with with storm right and the approach you're gonna have your speed layer and your bachelor anybody read this big data book that came out a couple years ago it's a good book you know things have changed though we don't need to have those approaches anymore now you see people using the cap architecture and stuff people like the kafka creators with Sansa the gear pump creators at Intel you know spark streaming flank all of which except playing sort of are built with Scala anyway so and you can always replace at state right that's fantastic so you don't have the idea that I have a single view of my state right now right I always know what happened that led me here and that's important you can always replace at let's see we have our persistent entity that can be an aggregate route babe I cares but if you're reading the documentation you will see this inside the logo of documentation and if you really want to use a crud database behind it you can I mean we're not sitting there and saying this is the only way to build things oh yeah we are but we are we're saying if you need to do it that way you're just going to have to go through us to get to it right because this is the way we think system should be built now the thing I'm actually super excited about it sounds like the lamest part of it and that's the development experience how many people enjoy building restful services on the JVM you're lying it stinks it's horrible it's no fun you got to sit there and write your code you wore it up then you got to put it somewhere for tom cat or jetty maybe it gets redeployed through something like jay rebel right but either way it's just not a fun experience it's no fun at all what if just like in play you know play was optimized for the user experience such that if you had a web interface you could just refresh your browser it we you know compiled your code showed you the errors in the browser right saved you all this time but taking the war and moving into the right place inside the container I mean all that stuff just stinks play took it away we're doing that now for the Micra service and that does mean you're going to have to use SBT I'm sorry all right it's an unfortunate we we had so many discussions inside the company about how we were going to make this work for maven and we could but we would never have this development experience and that's the unfortunate thing about this we don't want to be a build company we want xvt to be awesome but the reason is we're depending on it to make things like play and log on you know fun relatively fun right so you sit there and you change an end point you change some code you save it it automatically redeploys that in your development environment for testing you don't have to do any of that and that's including multiple services all aggregated it's a single grouping that's amazing so and it provides you than the in-memory you know Cassandra to run with and play with it's got a service locator which is enabling communications between these services defining the you know the physical locations and the ports and everything so they can be called it's also got a service gateway which is managing the proxying out to all of them and that overload mode for being able to redeploy quickly this is the fun stuff as fun as any of this is ever going to be anyway I'll be the first to admit and then of course there's the production environment and this is something that we have made part of our paid thing and that doesn't mean you have to use our tools to do it you can use other tools there are plenty of them out there but we're wrapping things for simplicity first of all we have our idea of conductor conductor is an Orchestrator like uber Nettie's if you're going to run three instances of your order entry service or something like that well how are you going to make sure that gets deployed across your footprint and how are you going to make sure that if one of those nodes splits off because of a network split or has a disk failure or something goes away how are you going to make sure a third comes up somewhere else that's the sort of thing conductor does but it does it with a peer-to-peer gossiping protocol like Cassandra like rioc instead of the idea of zookeeper underneath because zookeeper does have a single point of failure built into it in its configuration where it has to get it from one place that is not a resilient solution and let's face it zookeepers everywhere right it is we also had something inside of it we call split-brain resolver whereas an application you can define what happens when the network splits anybody ever seen a network split that happens quite a bit actually I worked the juniper for a bit that's kind of fun and they only tell you something when they do and zookeepers around you've got the idea of a majority quorum right if I've got nine nodes out there my zookeeper cluster say I've got five and one part of the split and foreign the other the four go down the five stay up that's great we need that we can't have them both staying up accepting requests and you know getting into a place where they can't merge where they are right that's the problem that nobody can solve so instead what happens if you have a 3 33 split what does zookeeper do down right because they only had one policy in a split brain resolver you have for you can choose whether you want to be oldest node in the cluster dictates which part of the cluster stades up you can have a referee node where you're dictating specifically only one depends different applications have different domain specific reasons for why they want these specific semantics quorum base majority based there's four different ways to choose it and you can do it for each application we have monitoring we didn't do monitoring really well at first I don't know if everybody remembers typesafe monitor console back in you know four years ago yeah we learned a really valuable lesson about that a we're not really good at UI design but B we learned something about actors if you're going to create millions of them it's really hard to capture all the information about every one of their mailboxes it doesn't scale so instead monitoring now is about what do you want to watch okay turn that on to and you just choose how much of a price you want this monitoring to cost you right that's a big deal and then the idea of scaling it out there having a single command to say put it out there three instances of my inventory service done all taken care of for you so you don't have to sit there and write all these scripts and maintain a proprietary pipeline of work to get stuff out there so everything we've done is orthogonal to the actual framework itself the idea being yes you can do this on your own but if you want to use Cooper Nettie's good luck a zookeeper be good luck using at CD right all the tools you have to wrap around qu benetti's to make this thing work and then there's always the question of is it so awesome why is Google only sort of using this thing yet right they're not really using it themselves hopeful as Eric Brewer pointed out they're going to move in that direction because that'll help harden the experience for everybody around goober Nettie's that has to happen kuber Nettie's is an important tool right it's the open-source one ours is part of our platform so test locally and then just push it to production very simple launched multiple instances with a single command sounds great right i'm going to show you an example as soon as I finish this but first i want to show you links there's the project site itself right now it's under light Bend calm / logo and then there's the github repo github com / logo and underneath that log um not repo but group name whatever you call it in github there's also the project that i'm going to show you called chirper and you can just you know clone the project and run SPT and run it locally it works I know because I did it last night and then there's the documentation itself which you can find a link to through the log on site but either way that explains all of the philosophy were espousing here log um it's about philosophy it's not so much about technology our belief of how these systems should be architecture so real quickly here so I'm going to burn your eyeballs for a second here and I'm going to show USB key an SVT build SBT showing you how a project is aggregated right and in this case I'm just saying that got some organization and I'm going to have a whole bunch of services inside of here and yes I had to define a scholar version for a java project that's SBT for you anyway i define all of these different services and i say their dependencies between them but once I've done that now all I have to do is to wayward can't really see this the beautiful thing about this is i just start SVT that'll be quick hi alright so eventually it's going to load up the project context it shouldn't have to get any new dependencies or anything like that it should be up to date and now when I say the run all command it's going to start the embedded Cassandra server it's going to start up each one of these services independently and in parallel it's going to register them and it's going to allow for proxying of all requests that need to go between the services such that you don't have to do anything work yourself so you see here it says the service locator is at localhost 8001 is where the service gateway is how am I going to make a request to this overall deployment of a service and then warning node.js detection fail who cares eventually it's going to say that I've got all my services started it's doing this in parallel on a very slow macbook air but now there's out there right you see they have these ports and every time it redeploys this locally to be using the same ports by the way it's keeping track of this stuff and now if I want to I can go to localhost 9000 and pull tripper up and it's using a bunch of different services for me to be able to have like a Twitter experience if you just go and look at the logon page you can see a video about this I don't want to sit here and try and create accounts and you know show tweets it's a hoot cares big deal right but it works you can play around with this yourself you see the ideals that we've espoused inside of logo so all right we may be wrong about this by the way who cares we can't find out unless we try that's sort of the attitude that light Bend has so here we had to start um no who cares i just had a question slide questions anybody yeah we're doing streaming actually in this case we're using acha streams yeah yeah I'm sorry yeah good question is it using HTTP between these services no it's using haka streams in this case I'm not sure if it's using an HTTP based stream or was using another I don't know I haven't looked at myself so yeah yes question 15 may be providing a way of computing services on different languages if you try to put in a c-sharp API to develop your micro services so patient so the question is can I have different services to find that can all be part of this integration layer I don't know i don't think so i think in it and initially no but in time I could see why we want to write the idea to define maybe I've got to drop wizard service over here maybe I've got an oj s service over here maybe I've got it dotnet one over here but I can aggregate their deployment because conductor technically can do it we just don't sit around telling people the conductor can deploy postgres and all the other things in the world because there's only so many things we could test with right now so yeah yes we're using place for our web services simply by ignoring all the other stuff we don't use we can't play how hard would it be in court so the question is using play how hard would it be to port to log on are you ask Allah user probably excruciating because of the Java API right now for what it's worth log on really is a mask over play logon is a lot of play underneath the idea is to make an opinionated implementation instead of just saying here is a web framework so I think what we need to do is have a migration guide for people who want to move from a play app to a logo map question specifically regarding the I can't now suddenly forgetting that term everybody ya see qrs and events or event sourcing so if we want to do see urs we've got plus press this question number one number two the deployment conveniences that you offer seem to be what we're already doing with their who hasn't and elastic Beanstalk and doctor etc what do you have a combat absolute it's a great question so the first question is if I'm using postgres can I do things like c QR s an event sourcing and the answer is yes right there's no reason you can't postgres is ambivalent to how you're actually organizing your data you could be putting json in there right so yes absolutely you could use postgres with this how you segregate postgres i'm not sure such that you make sure that you do not have a shared schema right so as long as you can make sure your schemas are independent you could even have a single postgres datastore so long as things don't affect each other but here's the thing you want to distributed store behind this and therefore you have to think in terms of how you're going to make postgres run data that may be replicated across multiple nodes which is already a problem i'm sure you have this is why i tend to use things like his you know DynamoDB type stores dynamo stores as we call them cassandra DynamoDB react I love the semantics of distributed data in these cases right but they don't do every case really well like JSON I okay as you move into a micro service world it's going to get harder and you can embed this is one of the things I talked about my book effective acha where if I have an update in this case I really only talked about in reads but if I had an update to has to go to three different places right I can have an actor that represents the context of that particular you know thing that needs to be done and it can manage the fact that two out of three worked what do I do with the third do I we tried five times and then you know fail because it didn't work everything has to be bound in both time and space five times in a minute or something like that right those kind of semantics you can build in there and have a real transaction semantics things like the saga pattern right if you have that then you have the ability to withstand all the different kinds of failures and even have a specific behavior for each one as opposed to the blunt just roll back right the second part of your question was about you have a tool chain already we're using AWS using the elastic Beanstalk using docker docker you can use with this stuff it you can wrap it in docker we don't care we don't make you use docker and to be honest I'm torn about docker that's me I think that the people who write docker are raking you know trying really hard but a lot of them don't build systems for the enterprise and they make some mistakes that are questionable around security around versions of things that they pull in you saw the whole Oracle blow up this week well actually happened a year ago where they were pulling in a version of oracle that wasn't even released yet and saying that that was the version of Java that everybody should be using a docker it's pretty ugly you know little things like that they're trying hard they'll get better but right now it's difficult right containerization probably is the future containers orthogonal the idea of elastic Beanstalk not orthogonal very relevant but what we're doing instead is we're merging AWS auto scaling into conductor right and I'm talking about some vaporware right now because it's due out next month but the idea is that if you have auto scaling turned on how are you going to scale beyond the footprint you've already defined multiple images right in this case you have to have conductor up and running on these image and you can auto scale and you know built into conductor and not use all these different tools and also not be tied to the Amazon chain you'll be dyed ours yes yes it is such as the deployment model and important back to play it's all yeah the question was is and that's going to go back to play and all of it is you can use play with conductor you can use yeah absolutely yeah so it's just we're building it into law Gomes production deployment experience so that you don't have to think about it but in play you would have to make it a component external to play so yes questions there was one more as you correctly but free I'm sorry we did what was spray we did yeah yeah the question was do we integrate spray yes we have assimilated spray okay so lost of chicken done to spray was right I cannot six month chickens if you spray it up you made some no so the question is a spray abandoned no it's not abandoned is it legacy yes so the idea was we bought spray you know Matthias and his co-founder you know they had really great ideas but they were very Scala specific and we wanted to make this an acha project and if you make something like a project it's got to have a level of documentation and product izing and a java api which meant it had to be architected that's where acha HTTP came in acha HTTP is now i would say through i would say three quarters fully optimized at this point because it does do the request response now in a fully optimized way other semantics not optimized yet but if you're doing requests or sponsor man thix it is as fast as spray or just a little slower actually so that's great but spray it's something we have to maintain because we have customers who rely on spray they've got legacy applications deployed with it we will continue to maintain it it won't die if you have issues with it continue to post it on the github and as you know acha user group I think it's more active than spray user was now so yeah just seemed like well we can strive against the way yeah and that's why I'm then you cast aspersions on cash yes testing disparate aspersions on distributed cache about that how much can I say without getting in trouble haha okay yeah pardon yeah right I I had a slide by the way just before just so I could say this I had a slide but I took it out it was a certain large company there's a steward of Java saying micra services and j ee and showing all the things you can use to make microservice it was a nightmare right it's a whole je ecosystem trying to tell you this is how you build microservices and I can just see Reza Rahman sitting out there saying yeah you know no no so SOA Mike asked the question you know it's sort of if I can encapsulate what you said so far so they don't lose it SOA I've said a lot about things that resonate a lot with people who remember SOA and I have cast aspersions toward the whole distributed fabric data fabric world anymore if without acknowledging that you are you're right actually accept and what he said was Cassandra is actually doing a lot of that and that's sort of true cassandra is a distributed store with replication semantics that are not locked based or consistency based until you've specified quorums right you have to say if I've got five nodes in my Cassandra cluster I want to write to three of them before i say i've got a successful right I want to read from two of them to make sure I've got a successful read so that I have agreement in the values that I'm getting and Cassandra will manage that for you right but it's about the isolation between services that Cassandra should never be exposed to anybody else calling in they should have to go through the micro serve its front end to get to it that is the distinction between it in the data fabric and therefore you never have shared data views between multiple services where data fabrics tend to live right that I'm going to have all this data out there and everybody can just consume it it's always going to be consistent no doesn't work well people have been trying really hard did you guys ever use this by the way this is brian hannah fee over here he's the he's a really high level person at wells fargo and brian is also the editor for the book that Roland and I have been writing called reactor design patterns if you ask me he probably deserves his name on the book more than me and that's because I've sort of fallen off and he's he's doing a lot of the you know writing with Roland so if you guys used to date a fabric at your place we have my world back end not in front face and stuff you know okay we're using something like you know gemfire you know I know the jump fire people they're awesome really good engineers trying to solve an incredibly hard problem you know press away I think that a lot of the concepts we have here represented the right way to build systems that's a way wasn't a bad idea it just wasn't dieded well enough it didn't have enough of the rules around and say it the isolation right that we have to restrict how these services were implemented or at least nobody built them that way that's why you saw people out there talking about how you could build SOA with rails it's like oh god please I mean rails is great when you want to build something cute you know it works you know you can get something up there like rap genius hahaha but you know when you really need to scale suddenly you've got 300 dinos on Heroku running and you're paying thirty thousand dollars a month as a start-up just because that's the only way you could scale so yes well I'm special so he uses BTW doesn't mean that developers need to learn Scala for SBT so if you're using SBT do you have to know Scala to use SBT yes and no SBT is better than it was I'm not saying it's great I'm just saying it's better than was it has a DSL yes if you want to write any custom work inside of spt by god you're going to be writing scholar sorry well for the most part they're sitting there and writing these just descriptors of services question is what does a logon developer have to know to be able to write an S BTW profile for their application and for the most part there's actually a DSL over the SBT dsl in order to make it accessible for java developers as much as possible but unfortunately that doesn't mean that you get away from the idea that you've got are you close mason clips it's on one of my screen somewhere I don't know where either way it's meant to be a DSL that allows you to express things in a much cleaner way than having to get to the low-level SBT things when you do want to do anything it's gonna be like writing a plug-in yeah you're gonna have a bilge skyline it's gonna be scholar alright well I am anymore yes using the patterns so the question is what are the top five companies using the patterns in logo that's really interesting question well first of all I don't think anybody's using logo them in production yet but the patterns themselves the the ideals well i would say that comcast is one and i know because i worked there so if you're wondering who i was talking about earlier i guess i just said i could say that goldman sachs is and they were doing it even before they started moving to our platform by the way goldman sachs is so invested in scala now that they're one of the founding people involved with scala center including nitro Thank You nitro including you know verizon Thank You verizon and also who's the fourth one IBM how could I forget that my god yeah IBM huge you know it's just amazing it's so exciting to see this much uptake Goldman has been doing seek urs based semantics and isolation of services for some time well we've been pushing people but that doesn't mean that everybody always listens to us and that's okay I mean there are realities as a business sometimes that aren't always apparent and we're not a consulting firm we don't sit there and I say that is that a consulting right we're not a current team that's going to sit there and come in and work with you for 20 years and never leave we come in and we look at what you're doing we tell you things that we think you know you we don't like for scalability reads and single points of failure all that kind of stuff but you know whether you listen to us or not is another matter companies that are starting to do more that's right now pearson education pearson education in denver is now holistically going with this approach and they're finally this is amazing they recently like about six months ago gave somebody an award for having a service that didn't go down for six months it was the first time ever in their organization that they had something with six months up time and then another one is Nomura Nomura bank in Japan they have a trading system that's been built with acha using these principles been up for four years no down time yeah and the reason is distributed systems the semantics allow you to have no one place that's going to fail they do not allow for the ultimate and low latency because then you need locality when you need locality you're paying a cost in you know what you are all going to bundle together there to do work right so that's a tricky thing they get five okay well those those are companies who were big Scala center groups the the IBM verizon though verizon is doing a micro service based implementation at on cue now called go 90 I think the streaming video thing that they do and Netflix obviously follow these principles and Netflix actually uses a lot of skyline acha Atlas a new thing that they just pushed out a couple months ago built with acha matter of fact it got just left here who was on that team soon yeah all right well thank you