Scale By The Bay 2018: Dennis Adjei Baah, Scala the Cloud Native Way...

Scale By The Bay 2018: Dennis Adjei Baah, Scala the Cloud Native Way...

Recording: Scale By The Bay 2018: Dennis Adjei Baah, Scala the Cloud Native Way...

you also thank you thank you everyone for coming to my talk today I'll be talking about Lincoln II and some of the things that we learned when we deployed it in multiple cloud environments you may find this talk useful if you actually run Lincoln in production or if you're running any high concurrency JVM application in the cloud but before we get into it I'd like to share a little bit about myself kind of loosen up loosen myself up I am from a small southern African country called Botswana anyone know where that is nice nice but I was raised by my grandparents and you may be asking where we where am i sharing this well I have this weird thing that if someone asks me where I'm from depending on the day depending on how I'm feeling how either say I'm from Botswana or I'll say you're from Ghana and every time when they come back and ask me again or in a conversation people get really confused because I say something different so today I'm not gonna try and confuse everyone I'm gonna tell you that I'm from both countries hopefully that lessens the confusion anyways I'm a software engineer buoyant where I maintain lincolni the open source project if you want to check out some of the cool things that we're doing with link ID and a buoyant or if you want to ask any questions or if you want to send me memes that's pretty much what I do on the day on the times when I'm not working I just look at memes you can reach me at on Twitter or if you want to get these slides these slides are on my github page might get up handle is right here so what is link are you how many people are losing link early in production okay good so link ID is a service mesh that abstracts service to service communication into its own layer and what what that does is it provides you reliability security and visibility and lickity is built on finagle and niddy so it's able to handle a high amounts of concurrency able to handle large requests loads without breaking a sweat with the caveat that you give it ample amount of resources and will it's pretty much what the whole talk is about but we're going to get into that what's great about linka D is that it runs as a separate process and so you don't have to include any libraries or any any frameworks in your micro-services code just to get some of the benefits that it gives you so what does it give you well right off the bat gives you service discovery integration we currently support multiple service discovery backends thinking up to top of my head we've got console we've got marathon and Maysles we've got kubernetes DNS and many more and what's great about link ID is that it's it allows you to extend functionality through its plug-in system all you have to do is just build a jar that has that that conforms to lincolnís service discovery interface drop in that jar in its class path and link it is able to read that and apply that new functionality that you've created so on top of service discovery you also get load balancing the load balancing algorithm we all know all know and love round-robin but it also gives you additional load balancing algorithms that work off of observed latency and this kind of helps tame your latency values across your percentiles other things you can do are configure retries configure timeouts for those retries so so that they don't happen forever and liquidy has this concept of delegation tables or D tabs and what this does is allows you to add more complex routing for your services so that you can do things like canary deployments or do Bluegreen deployments when you are running Turing running things in production you've also got the security aspect of linker D where you can apply TLS forced for your services or you can apply TLS between linker D instances so we've got all these features but where did it all start well started in 2015 where we saw that finagle was a violin a go was a viable option for programmers to build highly concurrent services and applications and we really owe this to Scala because of his code expressiveness and its unique mix between functional programming and object-oriented programming and on top of that the fact that you can write Scala applications and then compile it down so that it runs on the bytecode we get the benefits of of the JVM and its interoperative interoperability and being able to move it wherever we need to move it in our environments fast forward to today Lincoln II has been in production and various many companies for about 2-3 years we have a very active community lots of contributions we see come through there and I'm really I've been working a point for a little over a year and I've been really excited and energized by seeing all the work that's coming through it right now look at these servicing billions and billions billions of requests which is kind of hard to imagine when you look at the dashboards but it's pretty cool to see and I'm really excited for where things go next all right so why are we talking about this well there's so over these over the three years there's been one thing that we've been fighting and that's memory our memory usage but why does that matter well over the past five years we've been seeing this migration from traditional architectures like our three layered architectures to the cloud and it's natural for Lincoln II or Lincoln II was designed purposely to accompany these micro services run in the cloud and the issue that we run into is that Lincoln team is built on the JVM the JVM is great it's got a rich rich ecosystem and it's designed to scale up very well and handle tons and tons and tons of traffic but when you want to do the opposite or you want to scale things down you have to do a great deal of work just to make that happen so what but why do we want to do that well when we design linka D was designed to work as a sidecar designed to work as a distributed proxy where it runs along each and every one of your services so that you can kind of do the things that we all want like the service discovery the tries and all that with the sidecar the sidecar approach you have more finer grained control of running your services and making changes to your services but again the memory usage is kind of a bottleneck you can you can you can do a lot to bring this down to about 100 megabytes running on your system but when you've got a hundred plus services and you have to deploy this for each and every service that is not that's not gonna fly you've got you end up having like a really huge cloud infrastructure bill and things people are not happy so we kind of went back to the drawing board we started thinking about okay so what can we do to mitigate this issue so all right well deploy a link ad4 deploy one instance of likud lincolni for each host and this is the per host deployment model that we recommend for anyone that is trying to use link ad with this you still get the service to service level kind of reliability security and all that but this also has its disadvantages as well right at the bat if you are really strict or if you have really top security priorities you can't really do mutual TLS because you have to do TLS between your lincolni instances you can't do it from one service to the other and then things like upgrading link ID become very difficult where if you want to upgrade just one link ad instance that's runny you have to basically give up service routing for all the pods or all the containers that are running on one host so what's it like to deploy a linker date today if you go to our github page get the executable and then run it this is something this is kind of what you get so in this example I have lincolni deployed in kubernetes this kubernetes cluster has three nodes so we've got one Lincoln II instance for each node and we can see we're almost reaching four hundred megabytes of memory consumption and this is at a thousand requests per second and so for the past three years we're like can we do better is this the status quo do we let people use this and so we went on this journey to optimize link Rd and so there's an overview of kind of what we did we did basically JVM tuning we did we started to play around with different kinds of VMs and then I'll talk about that this last point a little later so with the the tuning we started off with the JVM started missing with flags on the hotspot VM some of the things that we did we're like tons tons of things I had to do with garbage collection in job and JDK 8 we turned on concurrent mark-sweep we turned off tiered compilation that helped it helped a little bit and we started just twisting knobs and tweaking things and so once we had enough fun with that we decided ok what else can we what else can we tweak so we we started looking at finagle and it turns out finagle has a lot of options that you can configure so that you can handle different types of load handle different types of requests and the ones that we worked with particularly where the threat was was the thread those handling threads or settings for threads so that we could control concurrency and also we configured memory used for each thread so that we could tame memory technically needed and it turns out for nettie also kind of has some of these settings but after doing all this we found that there was no magic configuration no magical set of parameters that would make lincolni Lickety run perfectly for everyone because depending on your system depending on how many clusters that you have running or how many knows that you have running in your environment things never stay the same so we we started to ask what is one thing that we can change that is not difficult to do but will bring the biggest impact and that's when we came across open j9 and open j9 is a VM built the IBM java development team and it's been tested for decades they've been using it for a really long time but they just recently open-source to it I don't remember the actual date I think it may have been 2017 and when you go through the page go through the documentation they say that it's a VM design for the cloud and so we're like okay alright let's hear what they have to say and so they have some benchmarks that they post on their site and basically what it is it's a small application like a day trading app that is given about a gigabyte of memory and sent different types of requests loads and basically just trying to compare different scenarios for that app they run these tests with open JDK run it with open j9 the open j9 VM and then another version of the open j9v em with some optimization and the results were actually pretty impressive what they saw was a 66 percent reduction in memory footprint and 40% not increased decrease in start-up time that would be bad if we have that and so we were like ok these are pretty good what happens if we add it to link ID and so this is what we had before almost hitting about 400 megabytes and then now we're able to bring this down to about 213 and it was really funny when I was working through this because I didn't believe it I was like I must have had something wrong so I kept on refreshing my tests and I kept going getting the same result was like okay I guess that's what it is but this is just changing just changing the VM we didn't have to tweak anything we didn't have to do any additional any additional parameters to get these values if you do want to take this even further the open j9 VM has additional optimization flags that you can configure my favorite one is this virtualized this X tune virtualized one basically you can tell the VM that hey you're running in a cloud environment so I want you to be able to use threads efficiently VM does its thing does its analysis and it actually does a pretty good job at not using too many threads or not using too many too much of the CPU and this saves costs so if you want to save some money on your cloud environments turning that on is pretty useful okay so this is link ID with a bunch of optimization flags added we're able to get this down to 172 so still kind of almost at that 200 mark but we're doing pretty good this is 170 170 megabytes per host that's that's that's acceptable for some and so we continued our quest to look for the one great VM and then we kind of found growl VM and I'm sure you probably have heard other people talking about growl VM it's a VM that is a VM that's aimed at breaking down boundaries between languages where you can compile everything down to one single kind of bytecode and running run it on one VM and what we were looking for in growl was the ahead of time compilation and also the ability to build native images because we wanted to be able to run linker D without having to have this this huge bloated VM running alongside it and what was interesting about growl was the closed world assumption where you you can do ahead of time compilation and then get a highly optimized version of your application we started running into issues with this because linker D was designed to be pluggable so pretty much it's just a big application full of dynamic class loading libraries and so we kind of started or a struggle with that a little bit so we have a working group with a lot of people from Oracle kind of helping us you know helping us figure out how to get Lincoln you're running with crawl there's some work already been done where we are able to compile finagle and lady with growl with some slight code modifications and we're making a little bit of progress on linker D but as this this this is basically the heart of linker D where we're just loading in a bunch of libraries and this is kind of like our this has been our pain point that has kind of slowed things down but we're still continuing to work on this and hopefully we'll get to a place where we can run linka D on growl if you're interested in joining our working group we have a mailing list that we use to update people in this working group or we have our linker D slack Channel or channel specifically for growl that you can hop on and ask any questions if you need to okay so we've we've tweaked everything that we had we could tweak we've changed the JVM what else can we do well we started to think about this a little bit and we're like if we don't use the JVM now I realize that I'm at scale by the bay and this entire conference is pretty much about the JVM but before you bring out your pitchforks just let me explain myself for linker D there are two things that are really important to us and one is that we are not using too many system resources on your system and that we're adding the lowest amount of latency when we're approximately requests because we want to be able to be transparent we want you to go into your system and like not even think about linka D running not running in your system so these two these two requirements are pretty important to us and we're not kind of discrediting the JVM because it's definitely more than capable of handling high throughput and doing the things that we need to do in these scenarios but maybe we could do things a little differently and so that's where link 82 comes in with Lickety - we kind of we kind of rewrite everything we rewrite the proxy in rust we're able to get pretty very actually not pretty good but very good low memory footprints and also we're able to get sub millisecond latency and if you want to learn more about Lincoln t2 you can come talk to me after the talk so what are the what is the current state of liquidy right now we officially support OpenJDK with a bunch of JVM flags that we turn on for better performance lower memory footprint and we also have open j9 in kind of our arsenal releases so and we also would really love if you try out open j9 because the more people try it out the more bugs we find more issues we can tackle we're able to move this from experimental to generally available and then we're keeping track of work on growl VM seeing where that goes and while that's happening we're rewriting link 82 off of the JVM and this has been pretty expensive for us it's been using up a lot of our engineering cycles but we feel like this is probably the right thing to do and right now we don't have as much feature parity with a linker t1 but we're gonna get there we're just making sure that we are doing things carefully so the more people try liquidy - the more use cases that you have for us the more I guess development we're gonna see on the gritty - so just as a recap cloud native poses some challenges when you're working with the JVM the JVM is great I didn't come here to bash bash to bash the JVM it's got a reach rich ecosystem and the reason why buoyant and lincolni is here is because of that but we're also making efforts to move off the JVM just to see if our hypothesis is right and then at the end of the day this is this is all implementation detail it doesn't really matter how we get things done where it kind of does because we're all engineers but what's most important is that we're building linka D or we're building these these these applications to solve real-world problems to make the world a better place as cliche as that sounds if you want to stay connected you can we're going to be at pucon giving a lot of talks a lot of workshops on how to use Lync ready if you want to join our mailing list you can do so there if you want to join our slack we're always there to answer your questions and then a Twitter handle and they might get off handle my slides will be available there and that is my talk if you have any questions if so unlikely to be far enough down that path to have concluded sir uh i think i think we have a pretty good understanding of where we want to go and we are kind of going full steam ahead on that yeah envoy oh they're they're good [Laughter] they're great no I mean yeah Lincoln D and AH were kind of the same thing we have ideas of how we can make the service mesh better and they have their ideas and we kind of just learn from each other and see who's gonna get there yes [Music] I would say for the open j9 stuff I think it's really it's really good to try out even if you don't have any some of those strict constraints but if it ain't broke don't fix it you know if you want to stay on the hot spot VM that's that's fine by all of us but we kind of want to see where the open j9 stuff is gonna take us next so trying that out is pretty useful [Applause]