SBTB23:Kerim Satirli, Gaming at Scale: What we learned from building our own multiplayer environment
[Music] let me know if the audio is uh good enough raise your hand it's very important after lunch know was heavy but if you can hear me I I see some Smiles so uh either you're just uh along for the standup show or you're here for the tech talk uh either way welcome my name is kareim Sly I'm a senior developer Advocate at Hashi Corp or F Fus on telling people about best practices how to implement them and don't get paged when stuff breaks which is my favorite part of the job it's kind of like platform Engineering in that sense if you have thoughts about this talk I'm usually at Cas pretty much everywhere except on Blue Sky because they have a very different format so uh you want it closer even closer wow we got this there we go so real talk Tech conferences are great but we all have a personal life uh we had a pretty significant change last year uh we got a daughter uh so first time father best as I know and this is impactful right if you have kids it changes your life in the best possible way uh no more disposable income no more time to game um yeah and other than that it's also great because you're constantly sick but the people that laugh they know what I'm talking about but there's this promised future of them getting to the age where they can join you for gaming and so as an engineer of course I have to be ready for that I have to have a multiplayer gaming setup that we can use when we're ready in Li of um having enough kids right now to solve that with our team um sorry with our family I just enlisted my team we're all pretty big uh Minecraft players um by which I don't mean that we're good just mean that we play it a lot and this brings us to our first challenge um I have less time to game I want to use that time as Pro as properly as possible as well as possible you know like not waste it on on the stuff that you don't want to do and like most Engineers I don't enjoy doing the same tasks over and over again unless it's clicking rerun on your GitHub actions uh that's that's always a joy so the real problem here is how do we make the time to dig up hundreds of blocks thousands of blocks that we can use as Building Materials in Minecraft without enabling cheat modes because cheats are no fun right well simple don't build alone always build together this goes as well for Minecraft as it does for your engineering career if you help other people build you're going to be a better Builder uh they'll teach you things that you you didn't know so these are life lessons didn't expect that when I started out Minecraft we solved it slightly differently Engineers right uh overthink the uh solution instead of asking a human being to join you we uh we built a bot and we took this way too seriously we gave it a conversational interface uh I know ver very much into Ai and everything this is not a kind of conversational interface where you told about could you do this and you know like what are your thoughts on building this kind of house this is more um sending an API request and getting a 200 back um it's one way communication but it's still communication and of course you know I work in infrastructure I was an SRE uh my bot has to have observability uh so we buil data dog into the system and we track what we do it's a very costly bot but it is worth it and you know for the Sci-Fi offense here uh we gave the bot oh there we go two of the three laws of robotics uh we didn't really care about the bot protecting itself we have workload orchestration for that if the bot crashes if the bot uh kills itself we have software for that make it more manageable bit of code packages bought up it's a node.js application uh so I think yesterday somebody mentioned one of 40 gig containers we're only clocking in a two uh because node is gracious like that but it's great it works we can ship it around the world and um play together and once the bot is packaged we find ourselves in that situation well I have the bot I have the game uh I've got pretty good Hardware but do I want to run the bot locally of course not uh I mean who does that anymore they also have almost almost unlimited AWS uh credits uh thanks to my job so we may have over engineered this a little bit if you're thinking about um The Martian right now Mike Watney going you know I have to science the out of this PL this planet it's pretty much what we did uh for this demo we picked AWS because it's just more fun um to do it with one Cloud right now and also because our finance team was very insistent that I don't run this demo many more times I'll let you guess while I give the talk how much the the setup that we have cost and whoever gets it right I'll ship you chocolate from the Netherlands so first things first we can run the bot somewhere but first we really need a server to run the bot on not using my local machine so I work at a company called Hashi Corb we make a tool called Nomad which is a workload orchestration tool throw a container at it a Java executable sorry a Java jar file or an executable or whatever you have and we'll run it couple things that are important here uh we obviously want to define a service and the ports it runs on we want to make sure our data gets properly stored once we start building a world I don't want to reconnect and lose everything there data persistence is important no matter if you're gaming or working with AI models and then of course we also want to specify an image the Minecraft Community is very very generous in terms of providing stuff like this um the image we picked here Community contribution with every feature and every toggle exposed which is perfect because as Engineers we love nothing more than defining around 254 variables and setting same defaults then well as you can see we have a handful of these uh defaults in here uh all the code here I'll make available afterwards if you want to play with it I'll give you the caveat of uh don't do it in everywhere in the exact same way and of course we mentioned it's a Java application inside a Docker container on a hypervisor on another hypervisor uh so we need some resource constraints 12 gigs is nice if you can go for 24 or 30 or 19 too you're going to have a better time so let's quickly run this and then once it gets scheduled we'll switch to a browser in a second we will have a life running Minecraft server and if you happen to have Minecraft on your machine the Java Edition you're welcome to join uh interact with my bots please know that if you strike them they will strike back and there is slightly more of them then you will be on your own so switch over here and service running very boring technology right like run what is it the 70 80 lines of code execute it through the server and all of a sudden you have this running machine this is how technology should be I say this not as a representative of the company I say this as somebody who got paged in the middle of the night somebody who does not want to ever be paged in the middle of the night or during the day if your technology is not boring you're doing it wrong if your technology is fancy complex and needs a doc docs page that has 20 30 pages of reading material it's a great white paper it's not going to be very user friendly which means humans will Mist will make mistakes if you can avoid that always good so if we switch in here well we see it running it's a little small on my screen kind of hoping uh it's good enough for you uh and if we switch it over 0 out of 256 which is absolutely correct because nobody's connected right now uh Java applications it takes a little while to load of course and this is a live server this is a live demo uh and so there we have it multiplayer gaming environment you can support 256 uh people in I want to say about 30 29 AWS regions but if we only deployed in one region that's that's boring so let's first check if this makes sense right when you're building stuff never start with just the end result never do all the steps especially if you're doing things at scale never do all the re region in one go let's Build It Up step by step uh so just like before we did with the server we're going to express our code uh our bot as code and make sure that shows up as well uh couple configurations here uh the docker image which we pull from the GitHub registry uh it's all a Apache 2 so feel free to use it feel free to uh improve it and definitely feel free to correct my code because it's been a while but it works and that's what important uh then this is my personal favorite running a bot running any application super easy but Bots are people right we want to know what they're called so we give them the very human readable name of the AWS availability Zone there stored in which works for me but doesn't care and if the bot cares and Strikes me well see what we said before it's a win-win for most of us except the bot that is unhappy and of course finally um express some of the resources we clock it at high usage these Bots need about 70 MHz so you could run this on Raspberry Pi um two zeros I wouldn't recommend it though Network window those is horrible and it's not great for uh constantly screaming information but run the same image well sorry run the same flow run the job and uh switch over here and if my screen refreshed correctly then you should be seeing the uh job overview of the server and the bot running and then what you'll actually see is we have three Bots running because one bot is fun right but I mean we've all read or seen iroot I suggest if you've only seen it you should definitely read it it's a different and better story three Bots is always better than one bot single AWS region could be a single gcp region um we've got code to do it in uh digital ocean as well but a single region is not ha this this is not AWS best practices this is not any company's best practice it needs to be ha so we need a second region in there but I mean come on while we're at it second region it's fun right let's go uh let's go a little bit more expensive here uh question really is Where Do We Go From Here do we go here add a couple of things in a couple new clouds uh each Cloud assigned to a different continent should we do that should we go even deeper it has so many options the largest version of this demo that we ran um had three clouds in 89 different regions well 89 different region identifiers technically geographically speaking it was only about 60 uh because Azure and AWS and gcp they are obviously in the same cities and this is fun when you have an incident like um gcp had with one of their data centers that got flooded in the summer you can still keep gaming this is important so how do we do that well let me um switch back to my editor real quick and I'll show you some terraform uh before I dive too deep into that quick show of hands anyone ever heard of terraform all right sweet uh we do fun things with licenses and infrastructure uh if you didn't get the first part that's cool so terraform is an infrastructure as code Tool uh which allows you to express what you want in your cloud in your data dog account in in pretty much any provider that you can imagine as code and that's important because at 3: in the morning not having to remember what you need to type but just having a piece of code that you can run is very non-expensive whereas humans having to remember all the exact steps of what you need to do it's very expensive I'll never call it cheap because any time you get Disturbed for an incident that is not cheap there's a mental health impact there a physical health impact so don't discount that ever even if you feel like the hero at 3:00 a.m. or 3:15 when you fixed it is not worth it prevention is way better than um having to d a fire afterwards so let's do some interactive Parts uh who here uses 8 us raise your hand right sweet quick guess how many regions do you think AWS has we're talking 10 15 20 100 not points of presence not CDN but actual AWS regions about 20 it's a good guess 22 42 man you have access to all the iso and top secret and othero partitions as well I like it we should talk I have a demo that we could run uh 22 is a default if you open a new account right now uh not all 22 are always enabled but those are the ones your account will have access to uh 9 if you click enable on everything and then if you add in golf cloud and a couple of others well we haven't run this demo yet uh it goes up significantly lot of fun problem is I one don't know all of them to I don't really want to know all of them like most Engineers I'm a lazy engineer which means I prefer my tools to do the thinking for me and so in terraform there's an API call that allows me to get all the active regions from a provider in this case AWS and if we run that then we get pretty much this result back that you're seeing on screen right now uh a list of all the the regions that are active for my account and if I go in there or somebody goes in there most likely somebody from our finance team who got frustrated that I Rand this around the world in literally every Zone and they disable something might code dynamically responds to that by actually terminating the stuff there so back to our editor take a little bit of terraform code I'm showing you the highlights because obviously there's more than uh the 10 lines of code but the key here is we're taking the input of that region which you can see on um line two in the 4 each we're popping in all the uh the region names and then creating a different terraform deployment in all of those regions and and we attach that to a VCS repository that has the deployment for that region in terms of AWS we're talking about an autoscaling group that sets up um three clients three servers uh has some user data in there to install the right software and then pretty much does that and reports it back so we can do some orchestration the important part here is when you run this demo start out with the inactive Branch because that will not deploy anything that cost you money in all those regions if you don't your AWS um Bill well not the bill view will look like let me uh switch this over real quick for you um there we go and let me highlight this for you will uh result in this uh each of those is about you should be seeing 168 instances that are currently running uh each of those has a cost attached it this does not get covered by the free tier in case you were wondering but that's okay gaming is fun and you know maybe we'll turn this into a career that doesn't work guess we'll just talk about it it's also fun uh so the inactive Ranch is something we find out when uh when we ran this at scale the last time and realized that actually the deployment was fine it was very reative I we were able to add new regions new accounts even we just didn't have a good way of cleaning it up um it's costly having to do this all manually even with um tooling like AWS nuke not the best part and so as with any engineering team we had somebody from our Network team join us and I was like how do you do all the cers well we didn't want to we wanted to use default resources but of course uh it's a regional deployment you you kind of don't want to run everything across the public internet if you can prevent that uh because again cost and also just bad practice never run stuff across the public internet if you don't have to even if you have certificates and it's also much cheaper to um computationally cheaper and latency cheaper um to not uh so pretty good factors uh we ended up creating let me show you this one uh let me put it this way if you think a multicloud naming scheme is hard try coming up a cyer range scheme that doesn't Collide and works and is predictable across multiple clouds uh for us we parse the region string we break it down into Geo so um the continent they're in uh cardinality um so the directions of the compass or in the case of AWS because they also throw in the word central which is not part of your compass uh and of course um the index and that gives you something that looks like this uh We've tested this across all accounts except AWS China uh because we could not get access to that but it works hasn't collided yet and it definitely scales for the next 15 minutes and that's all we need and so if we run this I could run this on my CLI I'm running it in one of our commercial products because it's easier and Visually more pleasing to me to see it we get something that looks like this uh 28 29 workspaces that all deployed the same exact thing and so let me um show you how this really looks like for which there we go so if this worked out you should be seeing my Minecraft interface and while we wait for the Wi-Fi to decide if um the server is actually available we'll uh we'll just make up some stuff like the beautiful logo my designer came up with Minecraft defying um conference logos is is apparently a skill we'll just connect and hey there we go beautiful so uh first of all Minecraft server we deployed that it works uh we're in front of a Chase Building so this is going to be guess we're going to talk to a representative there later uh let's see if we can find our Bots uh so well there we go all uh what is that 47 I guess joined successfully so that's that's a great start and I think I see them down here somewhere which is going to be real creepy if they're actually there oh yes no kind of look hey there we go hey friends uh so U minecraft doesn't do a lot of collision detection as you can tell um these are actually 47 of them uh so let me go somewhere else they like following me which is nice um and then maybe not a straight line we also see we have one of the Bots um misbehaving all right there we go so because they're all standing in the exact same spot and their movement speed is surprisingly the exact same they all uh move as one until they uh get to a part where they no longer uh move as one so what we have here is the quickest way to um get points with your credit card if that's what you tied to your AWS account or an easy way to uh to build an army of bots I like to think of it more as the second part so uh now if we tell them to uh mine mine something mine dirt here we go so while they um go and destroy my uh my level uh let me uh switch back to my slides real quick there we go uh Bots are gone you should be seeing a pink uh screen pinkish Apple wants to call it pink so we're good we'll we'll go with that Bots are doing their stuff we have a server running you noticed one of the Bots was uh leaving and rejoining that's by Design uh it's not a bug it's it's actually part of the demo uh and honestly if you weren't paying attention you might think that this talk was about Minecraft but it wasn't in reality and and this is really what's key um this talk is about showing you how to build a system that's fall tolerant um across multiple regions multiple sorry multiple regions of the same provider multiple Cloud providers and then of course figuring out how to operate a system of that scale um even if you lose several regions you might lose several regions because somebody doesn't understand why you're deploying stuff in um Frankfurt right now you could be losing a region because there's a weather system that's affecting your Cloud which is always a fun one to put in an incident report and it could be that you're just trying to deploy a type of instance or setup that is currently exhausted um I think uh Chris from firan was talking about it yesterday um if you buy up enough GPU with AWS in certain regions especially in Europe right now GPU will be gone and that instance type will be reported by the API is no longer available it's a problem uh you need a system that is able to respond to that and this is one way of codifying that pattern one way of building a system that allows you to respond to those events um and of course when you codify your infrastructure especially when you make one region feel like another region you don't have the cognitive cost of trying to figure out well why is my deployment in Brazil so much different than the one in London it shouldn't be so uh three minutes ago couple of takeaways uh train like you play I think this one is is very important do not cut Corners in favor of delivering quicker it will cost you in the long run and it's not worth it it doesn't scale now because you will have to remember what you did and you will then have to do it again to verify that what you remembered is correct and then codify it and it becomes an absolute nightmare always uh especially when you work across multiple clouds multiple providers and everything um codify not just your infrastructure but also your processes we had a lot of fun with this uh especially when we built it the first time we've gotten it to the point where spinning up 178 CPU sorry 178 instance uh 168 instance demo takes us 5 minutes uh that is in large part because everything is codified we need one access key at least we actually do them by region so in case something leaks there's um blast radius limitation but you could do it with one access key just please don't use your road key AWS will get really upset and when an outage occurs there's nothing for us to worry about because we know how to easily get it up in a different region and then I think this this is really the key here uh when you build stuff architect for awareness let your system be enhanced by the knowledge of hey there's another region I could shift work to there's another region I can benefit from maybe that region is closer to your end customer but don't depend on it clust I showed you earlier works best when all three Cloud providers are running at the same time we have a couple thousand gahz obviously um available couple terabytes of uh memory I've never actually looked at the full amount and it's fun uh the a AWS part alone cost us 1,700 bucks for a 3-day demo uh but it was worth it because we had so much fun playing and we spun up a ton of bots so if you run this code please don't don't tell me that I didn't tell you I think it's also a note in the um in the read me but build a system that's aware of its surroundings take away a cloud provider see what happens and that's a hard one right chaos engineering is is one costly two it's very hard to defend to people and three it forces you to revisit what you're doing uh Gwen made a good call here work with your vendors not saying ask us I'm saying with your cloud provider ask them hey we want to do this what's the best approach to doing this how do we solve this problem that might be new for your company but is definitely not going to be new for that vendor they see it across a 100 a thousand maybe a million customers they'll have some some ideas things work better together but they shouldn't necessarily break if they are Standalone um and then finally if you want to check the code I use to build this pattern there's about 25 different repositories that go into this uh head on over to github.com slork loads uh we maintain a complete codebase it's all a too uh and has many more patterns in there got some uh got a QR code for you as well if that's easier and then I think that's it so uh somebody asked before slides will be available on speaker deck but really the slides they're just Minecraft screenshots the real thing you should get is the code check it out and let us know how it works for you thank you all