SF Scala: Jason Swartz, Scala & Functional Microservices
Recording: SF Scala: Jason Swartz, Scala & Functional Microservices
[Music] hey everyone so I'm going to talk about Scala and functional microservices yeah it's right over here all right that works oh hey everyone and Jason Schwartz work at twitch right now and I've don't wanna work at eBay and Netflix on writing services and micro services and want to share a little bit about how to make them functional because you know we're Scala fans and we like functional programming so I've been doing functional programming for a while I worked on a book and really got into it I've been Scala by the bay several times and sf Scala for a while and it's great to be back here again ok so that's what we're doing today the functional micro services start out ok I used to have like the what part of this but I think everyone's heard of it so I don't want to repeat it for everyone micro services you know 2016 is calling and saying micro services so that was big at the time the Y parts still really interesting core thing is a lot of people kind of make the move from monoliths monoliths are great Ruby on Rails there's Python class PHP they're a great way to get your business up and going quickly I can't think of a single place I've been which didn't have a mom with which made it successful which you then now have to fragment at twitch they started with rails moved to go Lane they kind of moved everything over about three years ago last month we finally removed the last bits of some nginx things and shut down rails finally I mean it's a long tail it's a lot to remove but there's a lot of good question to ask when you make that move I mean monoliths are you know it sounds bad but you know they're not they're not terrible I mean so let's say you have a bond length you have it in it could be in play it could be in PHP Python and you're thinking oh there's so much code this is a problem I'm gonna go ahead and fix this well the mum with itself isn't that bad I mean all the codes in one place that part's nice right you could use your IDE and you'll find all the code that exists it's easier to talk to each other you're not gonna have issues with one part of the code talking to another and you know when you deploy it boom all of your code is deployed so you know hey Monell is alright and that's the old talk that's pretty good thanks everyone that's keeping it short of course the problem is it's a little bit hard to scale them even if you have a whole box I mean it's a lot of stuff you can't just say I'm gonna deploy one part of your image deploy all your code at once and that third parts the most interesting it's the people pretty soon once you start to sculpt your engineering organization you find it doesn't work out so well to have a lot of people in one code base it gets a little bit tricky who's seen this before alright who knows what this is called yeah this is Conway's law so it's pretty awesome you know your organization's gonna produce something kind of matches how you communicate classical thing is if you're an engineering company and let's say you sell a compiler as a product you know for engineering teams probably you're going to deliver a four pass compiler seriously because it's going to match your structure their hard part is if you start having a mom if you have monolith in your startup your organization's a monolith if you start having a lot of engineering teams and your team structure doesn't really match the software that you have that's the huge issue and that's why I usually want to break things up into micro services so now people can own their own different their own parts of the code okay so let's talk about this a bit micro services micro services oh yeah people forgot that part ok so micro services I think makes sense but you have to figure out if it makes sense for your group do people know how to deploy them can they think in terms of them I mean I would never start with micro services I'd start out with monoliths you know build a bat massive play framework you know build a big app get some money get a business going get successful then when you start scaling saying you know ok we should break up because there's a lot of challenges doing that you have to make sure you don't actually accidentally take down the business or something so I want to start about talk about a sample application so we all know JIRA and other issue things right alright so that's a basic model let's say we have a monolith it's called tasks and it everything to do with task management and it writes to a database I didn't come up with this as I think previous speaker said we've been doing this for 25 years okay that's pretty that's pretty straightforward I think we all know about that this kind of works though you know with one team when you start getting if you want to break it up a little bit and say hey we're gonna do something like JIRA and we have one service that manages the issues and you store it in one place you know if one place it manages the users put that another place now it's like oh okay that'll kind of work that's like that's good for you know two teams that works pretty well and then as you start to work with it you can figure out okay we should design our services as we design our group you can either change start splitting apart your micro your monolith into micro services to match your team of course you can also reorganize your teams to match the software anyone heard of that it's called the it's actually called the inverse Conway maneuver and how people do it and then they're surprised to find there's a name for it yeah you move the people around to match the software either way I think it's really nice just to think about this to say how can we be effective and kind of own our own parts of the software anyways if you doing micro services the name micro parts kind of obvious hello know what I'm hitting the wrong thing that pride isn't help you at all were you trying to read this it's nice you kind of split the data you don't have to have one big place another common feature is you know they're kind of isolated the idea is if one thing goes down maybe the other one should still be up you don't take the whole service down just because there's one bad egg they should be resilient right also nice to have same thing it's you should have them pretty small I should have private data hey look it's the same thing as the benefits so you don't always get this it's nice to have this I think these are not you don't always have this with my core services I think this is what you should try to shoot for if you can have this if you can get there if you can keep them small if you make sure they're isolated I think what do people call it reactive similar thing reactive resilient if you can make sure that they're really strong that way I think then you're set all right so that's micro-services I understand against it keep talking about this now to really make this interesting who's done working service fishes it's anyone using like it's okay anyone using Sto or Envoy or all right well you know it's a really nice way to say okay now and we're going to deploy these how they find each other how do we talk to them do we have like a fixed load balancer do we know how they work as they scale do we have a good way to get access to each other findable monitor is scalable I think I have one or two others are resilient the R word come up spec and reactive who can tell me what a reactive microservices oh come on we've been hidden with this buzzword always user now that's pretty good you know we also we do have scalable and kind of you know to indicate they should be able to you should have a way to grow it it should be resilient you know they should be able to be by themselves isolated they should be able to you should be able to take one down any other still work and they should be generally reactive you know they shouldn't necessarily block they should be able to handle a lot of throughput they should really be able to work you know I guess a good way to say it is they should be able to react to any kind of traffic that you send to it anyways this is the nice to have some micro services any questions so far I'm going to tell you how to do it I just think it's nice to have I mean wow this would be great all right there's also a lot of tap there's also a lot of challenges if you then go to micro services for one thing instead of making local calls you now have to have one thing to talk to another you have to figure out the the line the throughput the latency you could have different data formats you do have to find a service you have to make sure it's up and keep track of that troubleshooting a monolith isn't actually so hard if you then have ten or twenty or fifty or like a hundred micro services like we have at which gets to be a little bit challenging to make sure you're on top of them all and you know a little bit more so I'm not saying Micra services are bad I'm saying they can be bad there's a lot of trade-offs monoliths are kind of odd I should have just said functional model isn't just that that's my talk now you know there are a lot of people who run into issues with micro services and keep splitting them up the more you know Microsoft is you have your morning the more you have a distributed system really important to keep on top of that all right all right I will tell you a little bit about how to build it so the hell part micro services in Scala now we're getting into it a good question who well okay you tend to want to get this wrapped up and put into production generally with Java's traditionally you can package the whole thing as a war for example packages as a jar you can bake am i you can put a machine image right now I can't docker images highlighted that's kind of the default way to do this who's deploying apps in docker format right now alright for people who aren't what what format do is to get your software to production but it doesn't work yeah I like Dockers myself I think the nice thing is people who tend to work with Dockers they can work what you're gonna have go Lane services you can have you know Java you're gonna see sharp you can really have so many options and your infrastructure is very unrelated to actually how you develop them I would recommend docker it's been great with mesos kubernetes CCS really taking off on that note a lot of different ways you can deploy this now a lot of people have continuous development well lumpy was CI to start with Jenkins you can build it you can click a button deploy it to production that's how we do in my group if you're willing to develop if you're really confident in your tests you know if your CI just deploy the whole thing to production nigh and you know maybe send you note hey by the way I deployed that stuff to production I hope that's okay continues to deployments what it's called really nice to get to that point vm's are still very popular VMs Amazon machine images Spinnaker's was a Netflix tool spun out it's really popular way to do Bluegreen deployment and it's actually very hooked in with kubernetes now docker deployment again very popular I'm using ECS on Amazon I find that works out really well a lot of people love Cooper Nettie's you have Cooper nice on Amazon now of course you get on Google who's using something that's not on here yeah it's kind of a big slide kind of covers it I kind of like ECS I like to have things deployed super fast we're able to find that we can get with our service about 150 instances deployed in about two minutes it's really nice too it's really amazing strategies you have core thing is when you're deploying micro services if you know given that you might have you know 10 or 20 or 50 getting the deployment rights really important you know gain your monolith update upload it isn't a big deal you get it done and that's all there is to do for their services you generally really want to make sure you don't have to don't create issues alright this is just kind of a breezy overview hey question should I deploy to that cloud yeah yeah totally Oracle because when I'm here at Oracle or Amazon or Google at this point it's getting harder and harder to justify not going to the cloud I'm not even gonna open this up to questions I'm just gonna be on the side of yes I'll be standing here later please come up and change my mind next thing what's it okay other people say oh sure I throw tested yeah I think that's awesome if you combine the we're gonna test the heck out my micro service with we're gonna have it automatically deployed at production because we're pretty darn confident in it I think that's a big win I'm all about automating the drudgery the testing part is super important generally you want to find out the issues before your customers do if your customers start telling you there's problems it's a little bit too late but you can also just come up and try to change my mind on that too I'm not going to go too much into this it's just that with microservices it starts to get a little complicated it is a distributed system some things to think about is how they're going to talk to each other how do they find each other if you deploy new services how do they get found are using DNS names do you have a smart way to keep on top of it if if you have 50 micro services and one of them goes down how can you tell how you can keep on top of it it is a it's a it's a challenging area it's why I was kidding around and saying hey model lists are actually pretty awesome I changed my mind it starts to get a little bit challenging with micro services it is a very it it is important I think to get these things right in fact I bet they have information on this at scale by the bay Alexi well I'm going to find out you know I think some of the actually the one I should have put on here envoy they do have some great service meshes now written in Scala I think envoy one that came no it's not envoy no not envoy linker D linker D came out of Twitter it's really amazing that they have Scala based service mesh I think it's worthwhile to keep track of these things it's important to get this right because services aren't necessarily live distributed systems all right so let's talk about the development part that part's fun I like the development of them a little bit about the framework who's using a scholar framework not on here yeah Oh what are you using oh very nice yes and back yeah I should probably put that on here plays getting plays continuing to be really really popular I've been happy with it although it does include so many things such as web template in you tend to build monoliths with it pretty quickly lagom is now has Scala I think for about the last year and a half I think it has really good support for event sourcing it's a very different model requires a little bit to get into akka HTTP wait did that become HTTP for us was that different okay that's and you know I hate I hate to say it here a test of Scala spring boots going to be more and more popular there are a lot of people who still think of your spring and they think XML and they think I've seen too many open brackets but it's it's nice the last spring developers very happy with it it's actually a very common way people started with Java and switch to Scala in the way alright enough about frameworks I think this part rule it gets really interesting I'm not really I know I have Postgres there involved I'm not really going to talk this up except that don't just choose one datastore I think it's important to find the right one I like Postgres a lot for reasons I'm going to get into shortly and Maria as well I think can be important to use any the right datastore that's available or more than one let me show you what I mean after I try to click my thing here all right I'm going to use crud here I think people know what that is all right I want to get into this I think it's important to talk about a me about crud and how you store things I'm gonna get a little bit of a deep dive into this so I never thought I'd do this but I did an animation on crud that's right crud great great update to the alright obviously you don't know what this is this means this is a basic for REST API so you want to create an create an item you want to read Knight it update and delete it seems obvious right it's pretty simple yeah all right that's good I mean I don't want to be too challenging here so we all do we do scholar on we do immutable data right we're used to saying okay if I have something it's not going to change if I the list about the data structure I can share it between different threads for example I can grab it from the future and it doesn't change and I have all these wonderful methods in the collections library which take advantage of first-class functions so I can work safely with immutable data but who's using the mutable tables you are Oh exactly yeah kind of add on to it yes Brett did you know that you can actually do needle bull data in Marina DB and Postgres you actually think of them as immutable data stores and I think yes whoa hey it's not part of the demo that's not mine either that message is more for my daughter to say hey this is daddy's alright thanks hate for the message but yeah I'm beautiful data we all know immutable collections for what the heck are immutable tables okay so let's say I have a test I'm in turn planting JIRA I have a tasks table I have hey is this done I have a title I have an assignee this schema makes sense right it's a relational table who are kind of familiar with that let's say I have a REST API want to update my issues a little bit I have a get forget to get always used to get one issue by D I can create an issue and I can update an issue does this make sense all right we're all pretty familiar with that question is if people call this what actually happens in the database and that's the functional part I want to take I want to get into here so at the top I have the actual call that's being made and I have the table that's being changed below so thing about this as an exercise let's say I do a post what do I get what happens in the database well I mean they'll if you do a post that's supposed to creep something right so it creates this row in the database okay great crowd know next slide all right so I posted what does it put do yeah so I just updated it all right look at that okay this kind of looks well and let's say the next step I do is a put again and I kind of changed done to true it's done true yeah so this this makes sense right all right yeah like it anything wrong yeah how well yeah but hard part is I mean if you look at the table down below it doesn't really look like it matches the events that came in does it I mean you kind of missed something you're like no way sick we had an assignee I'm not sure why that's null I mean if you just say if you don't even know what the items that came in are and you're just looking at this table and you're trying to bug it how will you construct the list of events in your mind it's a little bit challenging isn't it you know like well oh gee I forgot the lessons of shared global state and our database table is a shared global state and I don't actually know how we got there so it's a little bit troubling problem is if your table rows are mutable you're not going to have any history mutable table rows suffer from the same issue as like a Java ArrayList they have the same issues as immutable collections if they're mutable you lose track of how you got there if your table rows never change though you have history you have everything you need so I'm going to propose consider that you can change things in the database but each row should be immutable think of your table rows is like a mutable linked list you can add to the end but happen Ziff you never change a table room you'll have kind of a history you often change history of everything that happens which I think works out really well you're kind of locking down what happens at every point let's see this again okay I post I created a table entry killer I put oh I didn't change anything I just inserted a new rocker oh well that's kind of nice let's say if I put again haha there we are so the idea isn't really unique the idea is something that says this is that this is the entity ID not really the row ID and it looks like I have time stamp so I can tell what happened here so what happened at 19 after the hour could someone tell me well there's someone must have done something and someone must have called put and cleared that it wasn't me but at least I have a record now I know that at that time someone cleared that something happened came in maybe through a different access law human from somewhere but we have a pretty clear record now if we then added something like request IDs or x-rays or something or even a user ID we'd have a pretty good we have a pretty good track record of figuring out exactly what happened in this so this is a mutable table you know you can say hey kept one and you can have a nice query that says I'll give you the latest entry which has the entity ID that you want so to the caller they don't have to know that your table is immutable but you can actually do better there are ways to there are ways to build a append-only table this sounds like a joke doesn't it I mean but it's not good way to start mate let's say you have Postgres or Maria DP you want to make your tables immutable first thing you do is don't let your application to updates or deletes take that away next you can actually do that pretty simple next thing to do is have good columns to support an append-only table here's a good example here we have I D like the row ID which is actually the row ID but how now we have an issue ID which is an entity ID so I can have so the row ID is unique but issue ID I can have multiple copies and I can say oh not just created but I have a creator ID so I could see the user who did this so if we have someone who calls this and cleared out the assignee I don't have a whole history for every change that happened here and I can see who did it you don't have to do this I'm just saying this is a really nice way to have an append-only table and you can use this you can keep using your post graphs or Maria database and it'll all work really well ok now let's switch it up let's say just let's say you don't really need post Chris let's say you don't really need Maria you want to change it out we actually did just talk about events or scenes in the previous talk does anyone who's curious about what event sourcing is yours alright I can't dive in then so this is the immutable idea I think the things at the top are HTTP actions right there HDR api calls the stuff at the bottom is the state that happens as a result of your HTTP call i like to think the stuff at the top is an event you know you're saying create this and then update this then update this again the stuff at the bottom of state the funny thing is I could probably reproduce the bottom part just from the top part if I have a list of all the actions that happen I don't really I may not even need the state at the bottom so the event sourcing ideas let's just track the events and you could build the state from it the state is dependent I mean look I create an issue and the state is as she created and I assign it to someone and it's just an update and I complete it and it's marked as complete but you know the state's nice but I'm not sure I really need it States virtual I can build an 8-point basically I have a tape I can replay the tape from scratch and build out the side so the event sourcing says let's store the events let's actually store and persist them somewhere and then the state can be built at any time does event sourcing makes sense now all right and that's obviously you probably don't have enough to actually build something like this now it's just the basic concept but that is the kind of concept the event is the source not the state all right I just don't that here's an example one challenge though is what's the format of an event what are you storing in the event a lot of times with a at least dynamic languages a lot of people say well let's restore it let's have an event and just store it in Jason stored in the ammo and put that in the database or let's convert an object to some kind of textual form and convert it back let's use a relational table just store that there was an event and have something else to say okay we're going to marshal the if we're gonna unmarshal this back into something that's nice okay kind of move on I think we kind of covered that part yes real event virtual state pretty straightforward right yes sir common powder is to say will only store those events for a certain amount of time like it there are some places where you like if you have an order at Amazon it's probably only going to be live for a certain amount of time and then you can say we won't accept any more updates you can materialize that to a separate table and not worry about that I think event sourcing is more useful when you have events that are when you have more ephemeral data for longer lived items you're right it can be a pain because then you don't want to have to say oh let's figure out the state by rebuilding it from scratch faster systems will often use will often materialize this to temporary tables to say here's the state as we know it but say okay we got new events so we have to mark these mark this table as false it's probably a trade-off that depends on what kind of system you're building and if you're building something where the events will only come in at a certain amount of time grouped together or if you're going to have them over a larger time you yeah you probably want to start from some level in in video compression usually have keyframes which say here's the full frame and then after that you have a certain number of deaths based on it but you know at some point you say it's eventually it starts getting out of track and you just want to add a keyframe over a few items what's a comment is that the same as turning it off and on oh crap I thought this is new I figured next you're gonna tell me functional programming was not invented with Scala I I should have written more down in college but thank you for clarifying I thank you for having me clarifying all right the why parts kind of interesting I think it really depends in some places we found immutable tables really useful if we already have a regular system set up and we don't want to change everything last place on is that we use two met penalty tables for anything to do with Finance if people paid money if they had subscriptions we needed every version of it if you have people doing something you don't really care about you can probably update the tables nice thing is immutable append like tables can you know can live side by side with other information if you really want to make sure that you want to write stuff fast for example Cassandra is often used with the vent sourcing because you're appending stuff super quickly if you really want if you really are able to build your items so that the commands and writing in to commit the events in is very different from where you read it out the event sourcing could work this is just a short list of six I'm not saying this is a this is the easy way to organize it it's pretty useful though I can see how Oracle cloud would want to be you know have performance to be a major issue and design for this upfront any questions about an event sourcing yes sir you already filed a database that you're inserting immunity and you have the bed lock but that makes that meant longest sort of an event serves so or I happen it before I think it's highly dependent on the exact version of I seen my sequel I've done that too I've used it for like basic replication before I don't know I don't really trust that that I don't trust it's gonna have everything I need I know it has everything necessary for the database it's starting to get a bit low level I don't think they've been logs really created for that okay is that fair I mean I'd rather have tools for example if you if you have like a subscription record say hey here's my subscription to a gym and you have updates you may want to say to the user what's my current status but if you're calling customer support you may want them to see your whole history it's nice to be able to say show me the latest one versus show me the entire history of everything that's happened the updates to the bed log could be that other database that has yeah that's true I mean usually the usually the framework set up so that you kind of separate the command and the query and the commands you make sure I really put in order I mean usually these things are designed that way I don't know it could work to use the binlong but I like the idea of saying of event sourcing you know the events are immutable that part's nice right I mean some some systems you pull an events and say we don't allow that event or this is not in the right order but pretty much they don't change I mean it's pretty awesome you can play back you're saying hey I want to control the version of how this stuff changed this sounds like a certain three lever version control system right no oh man okay well we like all having control over this stuff we all use get we're very familiar with it it's nice to be able to say hey I okay I'll be specific yeah no sorry good close answer yeah there's three letters that Park was good but it's nice to be able to say I want some control and transparency into exactly what happens with my data I think event sourcing is a nice way to get there yeah I probably should update that even as of 26 2018 it's a little bit non-standard it's a little bit hard to understand and you really have to design for it I found again this is super useful for extremely high accuracy systems like financial trading I found it very useful for extremely high performance I'm not sure it's quite common as much I think it's just another good tool to know about I think that's the part of Microsoft development I was really going to get into yeah that's the whole thing I'm just kidding all right so that's everything that's what I want to talk about with functional micro services so this is great thanks for having me here today I'm Jason Schwartz has been kind of fun and thanks [Applause] you ever tried patching your Davis Ameritrade patching my database what do you mean I'd like a patch rather than a book or updating just changed oh yeah yeah put is a little bit weird put us where you're supposed to put are you talking about the HDP patch is that right yeah so supposedly patches when you want to say I want to update just one field put says here's the entire new version of the field for example so I actually usually use patch with things like examples like Gabri just update the assignee or just update one thing what you're supposed to give the whole record I probably use put here because it's easier to type the patch yeah traditionally it depends if put it is supposed to be for entire items but I like the idea of just updating one item at a time it gives you a nice history nice overview of exactly changed did I answer your question yeah I used an it include to hereby use window functions in Postgres but PostgreSQL supports two sequel 2003 which has window functions which you can say select every item in there but group by this entity ID and then just order them by the most recent first and then just give me the first one from each row Maria DB has sequel 2003 window functions money sequel 9 or I think Michael 10 does not have it them again it's a nice way to say hey I want to be able to query things but only take the most recent entry for a given entity but that way you know you people who call you know you're interested api's or graph qo api's don't really have to know how you have that implemented you could still they'll still get the right thing these queries are pretty fast they do hit the indices but it means you have the whole history and you have control over what people can see any other questions yes that's a super question how eventual consistency do you mean that they're actually put into different storage systems ok so I think you're saying what the problem is you might events coming into different parts of the system that eventually make it into the same storage but how do you trust what order they came in usually I trust the database itself I've used my sequel and Postgres to have multiple events coming in and only take the first one like typically anything I do with SNS or other venting queues I'll typically honor the first one by having a uniqueness constraint that says here's uniqueness constraint for the type of event coming in and the ID and the and a basic version of the originating timestamp and use the originating timestamp to say we'll only support the first entry here that's why I've seen that's a good question does anyone have ideas of how do I guess disintermediate or figure out the correct event to use when there's conflicts because there will be conflicts it is one of the issues you have with distributed systems I guess probably if there's if it's the say is it the same event or is it conflicting events [Music] certainiy repeat your question [Music] yes it's a great question I'm not familiar with patterns of how to do that for modern databases when I was an eBay we worked on end of auction code and there you really have to make sure you your honor the correctly we actually had business events in Oracle tables where when you inserted part of the query for inserting something was running a trigger that would go through and make sure there weren't anything competing items at the exact same time in the exact same table so we would chart based on the user ID based on the category of the item and then if you insert you don't just do an insert you do an insert and business processing event at the same time probably doing it at the database level is one way to make sure you don't have competing entries in there at the same time I hate to say it's locking but you keep that in you know and stored procedures and triggers are a little bit less common now but for something like a financial system or something where finances or auctions are at at stake I'd say try to char it and get to get it down so that you can actually have logic at the database level for that great question all right thanks everyone [Applause] you