scale.bythebay.io: Manish Pandit, Immutable AWS Deployments with Packer and Jenkins
Recording: scale.bythebay.io: Manish Pandit, Immutable AWS Deployments with Packer and Jenkins
I'm gonna walk around a little bit too and it's a small crowd so if you guys want to move to this side because I can probably move up to this pillar because of the camera so if you guys want to kind of move little here it's better unless you wanna sit there that's fine too I don't mic so we're going to talk about immutable AWS deployments are using packer and Jenkins so this talk is kind of more towards you know DevOps side of things a little bit so I'm gonna do a show of hands just to make sure I have a level set as we go through the talk but before we do that a little bit about myself so I'm Manish undeath and I work at Marietta it's a fin tech startup based out of Oakland I am the director of platform engineer in there so I am developer at heart but a DevOps card so so I've been kind of I'm in that DMZ area somewhere between engineering and DevOps which which I really love and in the way I am looking at how cloud is evolving in fact the talk before this is tear a lot of that work was in that whole DevOps side of things right because we write amazing code but what makes it even more amazing is how we deploy it right and that that matters especially at the cloud scale at which everything is going now so I'm on Twitter at lobster one two three four please if you want to follow me I do a lot of blog posts mostly around aw as some development stuff some DevOps stuff quite frequently you can follow my blog it's on it's on github and then when you follow me feel free to like DME or anything too if you have any questions either about this stock or about anything which you want to talk about regarding AWS or any lag doc so a show of hands how many of you deal with deployments oh good we got a pretty curated crowd here that's good that's good and then so is there like any tooling involved in your deployments quite quite a bit right I mean the days of SCP are gone so we have DevOps see a CD a lot of tooling in there and then how many of you are deploying on Amazon good what about the rest of you like any other cloud providers or pretty much it's either Amazon or private data center that should be fine you'll be able to relate a lot of these things to any other deployment platform which you have which is the beauty of Packer so what exactly is deployment deployment is essentially a process of pushing code beyond your development environment right we all love to code we all write a lot of code stick it in gear we do pull requests and do all the sweet stuff right but what happens after that that is all deployment like once the code has left your development environment it goes into the deployment domain and then deployments can be complete or they can be partial and there is a fine line because you know how deep can you go when it comes to complete by complete what I mean is you build a server from ground up that is complete but what is the ground is the ground hardware is the ground operating system is the ground application server or is the ground a container right so we are going to talk about that so how many of you when you deploy code are provisioning brand-new stuff from scratch and then putting your code in there so let's say like I have v1 running and two-run v2 I build brand new infrastructure and associate that with my load balancer load balancers the only thing which I am NOT rebuilding which by the way can change for Bluegreen testing you know based on which approach you use right so and then it can be partial partial is I do a lot of Jabbar stuff on the back end and the deployment artifact is called a war file right even in Scala so when you use like escalara project it spits out a war file right because at the end of the day you are going to deploy on a JVM container which is pretty much like Tomcat something like that so partial deployment would mean that I'm just over writing war files right which seems pretty simple pretty easy right so this is an example of of the complete deployment so you are provisioning this stack from ground up you know there's I'm here first then you're going to mount it and then you're going to install your applications applications in there and then you're gonna trigger it right so this is this kind of looks like complete but then it's like a lot of work right like every time I have to deploy a new version especially if I am doing Qunari deployments or something like that where I'm doing rolling deploys this becomes pretty time-consuming even if I were to automate it quite a bit because I'm doing everything from ground up right so partial is pretty common you know it's I admit it I've done it for a good chunk of my career and it just makes sense you know all I need is a war file why do I need to care about installing a new tomcat or installing like brand-new operating system and stuff like that you know who cares right I have a server running it's behind a load balancer I'll get a little bit of downtime don't get now supports on the fly hot deployments right so I just drop in the war file and it's gonna work because we trust the developer who wrote Tomcat so and it's gonna work right so we get this so I refer to it as in-place deployment because all you are doing is replacing the deployment artifact which could be a war file in case of Scala and Java or any JVM based frameworks or it could be anything else it could be an RPM for that matter so it gives like a false sense of speed and agility that hey you know all I do is drop drop and artefactum you're good to go so our typical steps are like this what I do is I a CP Dewar file to the production box and then I assess etch into it and I bounce Tomcat good right here the problem is that now I have credentials to SSH into the production box that's just one of many problems in this scenario but this is like very typical partial deployment another example is that I get a little advanced and I'm like you know what I'm not going to use a CP I'm going to w get and just grab the file instead of uploading it from my laptop I'm gonna I'm gonna take it a step further I'm gonna get it from Nexus right Nexus or artifactory any kind of artifact repository right it's used for versioning and stuff like that since you guys do so much deployment and kind of DevOps see things it's pretty typical scenario right your code your code gets checked in to get it triggers a Jenkins job that job runs tests once the tests are run it plus ones diversion then you cut a release that release goes to Nexus and then you have deploy man artifact right and then here what I'm doing is I'm doing W gate and then I'm just doing a restart again the same pretty much kind of the similar things as previous one except I'm not as seeping from my laptop and then I want to do automation on this right like how much how many W gets and SCPs I can do so I let me let me go ahead and automate this so I'm going to script this all I'm gonna run it through a Jenkins job and not only that there is also frameworks to do exactly this I've used fabric in the past it's a Python based framework which you can do to do exactly this you know it just runs remote so you can just say like remote W Garrett or a CPU this file bounds Tomcat remove temp folder and stuff like that a lot of cleanup so I've had some painful wounds from using using Python or not exactly fabric I'm sorry not fabric but it is that the wounds are not because of these frameworks the bones are because of in Europe of mutable deployments and there is Capistrano also so what could go wrong right and these are the three examples of the exact same cluster which you deploy in one case let's say I'm deploying Jenkins itself so in first case it worked and assuming that you know your load balance or health check is just TCP 22 or something like that right so if your health check is like you know slash health check or something like that best practice to tell the load balancer that your application is ready to serve traffic this would not happen but some symptom of this might still happen like could have different version so first example is it it works fine in the second case looks like something happened maybe Tomcat didn't come up clean enough or or we don't know right and in the third case the Dewar just did not deploy at all so Tomcat is up but my application is not there so something happened right and in order to troubleshoot this what I'll do is so these are like three servers behind a load balancer one works - don't - - do not I will SSH into these boxes and I'll go and I'll fix it a little bit and then it's going to start working till it stops working again right this is an example of a highly mutable deployment right you can clearly see there and then what I did was I put like these snowflakes in there so you will hear the storm a large snowflakes servers because every server looks different I could be running Tomcat 8 511 on this n8 513 on this because I am not touching it remember it's in place at all I care about is my war file and these are the symptoms which we run into and these are classic symptoms and you will start seeing this like every time you have to have 5 on the browser you'll see a different page or a different response so unpatched outdated dependencies no one is look taking a look at the operating system no one is taking a look at CentOS or RHEL or whatever I'm running to make sure whether it is it is running the latest and greatest is it security patched is everything in there what is supposed to be tomcat what about CVE right it's like vulnerability and exposures I think common vulnerability is an exposure CVE so that is like a security list which I think the government maintains and you to make sure that you know you don't end up like Equifax so you have to make sure that your packages are there they are deployed right so that there's some maintenance side of things no one cares about that in place deployment when we do inconsistent have behavior like I saw in the previous slide changes outside of deployment cycle what this means is that if something goes wrong it just SSH in the box and install something in there just to troubleshoot the server and that's it now you have made another snowflake out of this snowflake because now it's entirely different than other two right so you are increasing that Delta between the server's because for your customers your load balancer is your interface and it is a it is a fair assumption that all the instances behind a load balancer behave the same way right and in case of partial deployments or mutable deployments did not human errors right how many times have we gone in to in in that SSH sudo mode and mess things up I have a lot of times right so that increases the chances that I may have done something when last night I associated into the box and I probably I set up an agent which is filling up the log files and it's going to blow up tomorrow right so all of these things can happen and this model does not scale so when we are talking about rolling deploys when we are talking about blue/green deploys a lot of DS like manual deployments or even when you automate them to an extent they are not exactly going to scale so what is immutability right so immutability is pretty much an answer to all of this so what this means is that we build an entire runtime infrastructure from ground up and a nice I kind of if you remember a few minutes ago I kept saying infrastructure from ground up but let's qualify a little bit let's call it runtime infrastructure because we are not going to kind of you know put a motherboard in place and mount hard drives and SSDs and stuff like that so we are not going to do that that infrastructure is there but since dad does not change often like how many times will you upgrade the firmware of your SSD drives almost never right so let's leave that there that's a fair assumption to make that your Hardware underlying Hardware doesn't change your virtualization server does not change but everything beyond up is what we are going to talk about here right so build our entire runtime infrastructure from ground up right and runtime what do we call runtime as it is operating system all the libraries that you need and I I'd even go further and I'd say only the library is red you need app server this could be tomcat this could be anything which you are you know code of course to deployment artifact and agents the agent model is kind of it defeats the purpose of using SSH anymore right you can have agents for Splunk elke so you don't have to SSH into the boss just to look at the logs so you have log aggregators which do that performance monitoring there's no relic there's data all of these come with agents so you don't have to SSH into the box to find out how hard the CPU is right all of that is through agents so ideally you should not be using any SSH and I'll cover it in one of my slides it's like the immutable deployment Nirvana is when you are no longer when you can't even SSH into the instance no matter you could be super user you could be admin whatever the ami will or the instance will not allow SSH and once you're at a point where you do not need to SSH into the instance then you know that you have reached that ultimate goal so Amazon Web Services since that's why I did the show of hands so I because it's really unfair to cover Amazon in one slide so I'm going to still give it a shot so basically it is a collection of various services this is the best definition of Amazon I could come up with because it's it's huge it is huge and it's gonna get huger in two weeks at reinvent I'm sure they're gonna announce a lot of stuff I'm gonna be there by the way I'm giving a talk there as well on the 27th look forward to seeing you there if you guys are there please hit me up on Twitter and we'll meet up so compute so rate database messaging this just scratches the surface of Amazon and then it bills it helps build up build architectures that are highly available fault tolerant scalable and cost efficient based on how you use it basically know if you're running like X large instances then it's not cost efficient but if you want to be cost efficient gives you a lot of options to do that what our ami is how many of you are dealing with ami so when you say you're dealing with okay good good good so ami is basically think of it as an image and that's what it is it's a means machine image so it is a template to launch easy to instances now for those of you who are developers as well think of it as a class and an instance right class is basically it it defines what an instance of this class would look like right it has methods and it has data so it has state and it has state is the data and then it has functions to work on that data so similarly these easy these am is are essentially templates which define if I launch an instance using this am I what are those instance characteristics instance characteristics like what operating system is that in instance going to be on what is the virtualization type so Amazon supports to para virtual and h vm h vm is kind of referred one at least I prefer H VM and looks like Amazon is also kind of moving heavily towards h vm over a para virtualized storage type you know is it EBS bag what kind of storage options you have there or instance back volume attachments all of that stuff right and then it can be shared within accounts and this is this is pretty cool so in larger organizations the way you organize your Amazon is that you have multiple accounts you have an on product count of production account sometimes they create one account per org so IT will have its own account and engineering will have its own right so you can you can share a Mis between accounts and they can be private they can be even made public there is an Amazon Marketplace where there's a lot of em eyes you can go and buy those EMIs are pre-built with some servers and stuff right and then it is the highest level of deployment abstraction so when we are talking about immutable deployments we are going to talk about deploying the am is because the mid-50s slide right so we are right at the at we are not going rogue where we are doing in place deployments at the same time we are not going hardcore where we are starting with provisioning a hardware on its own right this is this seems like a good balance of where we should be in terms of humidity deployments so these are the a.m. eyes on on Amazon this is Amazon Linux by the way so I have seen a lot of installations on CentOS so CentOS has like a similar not a similar list because this list is managed by Amazon but if you go to launch an instance you can see all there is sues there is open to raise everything there but this is just the list of Amazonia mais they patch it release it pretty regularly and you can see here that it has the a mais are tied to region so you cannot like you cannot use you can copy them but you cannot if nem is try to region you cannot launch an instance in another region using that region say mi so you have to move them across regions if you have to so that's and then you can see the virtualization type there there is h vm there is a para virtual and based on you know what you want there's an EMI ID to get patched regularly and stuff like that right so what you do is you take this am I just a base ami from Amazon its Amazon Linux again not tying this to Amazon Linux it can use this the same thing for CentOS or any any Linux flavor that you want Windows I'm not too sure I've last time I used Windows was like mm I think so not too sure about how the windows world works but it should be similar there are windows em eyes ya know so mostly centers are open too so what you do is you trim the fat right the ami be the base am against comes with a bunch of stuff right it's a generic ami so it will have SSH it will have a whole bunch of libraries so what you do is you start removing all those things right you don't need a VI editor you don't need like Emacs you don't need a whole lot of stuff is there at am icons to it because you are building a server you are what you're building is a deployment artifact right you're not building like a desktop system right so what you do is you start trimming down the fact and then you configure the libraries and tune the parameters of this based on what you need from this application or from this server and then what you do is like you know make infrastructure not war that war is a is a java reference yeah so this is what we'll do right so what helps us do this is something called packer and Packer is a tool from Hoshi Corp Hoshi Corp is really they have phenomenal products when it comes to DevOps and CSC D and automation stuff like that vagrant is from them there is console we use console at marketer for service discovery quite a bit so console is also a she cart there is Walt there's a whole bunch of stuff coming out of Hoshi Corp how many of you have used Packer in the past you do perfect so Packer is essentially a tool which lets you provision and also configure these em eyes to perform immutable deployments and that is like one of the many tools from asha carp if you guys have not looked at Hoshi core website I strongly suggest check their products out they are all open source as far as I know and they are really really useful like vagrant and stuff like that you will find a lot of value in vagrant we use vagrant for a lot of our test environments where we make sure that our test environments look exactly like our production environments and development environment so vagrant is good for that many many tools so we are going to talk about Packer in this in this talk so it supports multiple providers supports multiple Provisionals now what is a provider so we are going to talk about provider but installation is really straightforward you can stall it via brew or you can just download it's just one file it doesn't come as a package and everything like that you know we just go and you just download it it's it's just Packer it has the command it downloads a specular zip and it explodes our spatter so it's 1.1.1 I think probably it is the latest and greatest and then what you need is when you are going to use Packer from f4 Amazon at least and again you know I'm trying to kind of keep the context to Amazon but same is true for any other provider you will need an ec2 role and essentially you're going to run it from your Jenkins server or wherever and that server would need to talk to Amazon so what you need to do is you need a EMI credentials Oh Amazon credentials I am credentials and those credentials will have a policy I am policy with them because you want to make sure that you're not running this as admin user you know you don't want to give it that much control right and then the details of that policy are available on the on the Packer website and if you are running it from Amazon you don't need to worry about credentials you will rely on the I am role to control packer so what are builders so you know Packer has builders and provisioners those two things so what is a builder so builder is basically it defines it defines machine images for multiple platforms and it supports a lot of platforms in fact there are few in there I didn't even know existed there was my first introduction to Alli cloud I had no idea Alibaba has a cloud - I saw that listed in there and I was like what is this early cloud and I saw that and why it's like awesome it's really good coming up so a lot of platforms are supported there you can if you use docker you're probably used to writing docker files a lot you can provision docker containers using Packer as well so you can have that uniformity uniformity in there and you can regenerate and stuff like that which generally with docker files it's a little complicated because docker versioning relies heavily on the tags versus here you can use the real version control popular AWS mi VMware docker Azure GCP even VirtualBox you can use the builders for that and you can even do custom builders if you want to so that framework exists so this is what a builder looks like this is a simple Packer template so I've used a ami ID for us east one I think and then these are the variables so it is going to either pick the variables from your credential file or it is going to rely on ec2 instance role if your running this on an ec2 instance in Amazon okay and then what I do is I specify what am I supposed to build here so there is Amazon EBS Amazon has an instance backed AMI and an EBS back DMI and again this talk will be too short if I go into details of that and there's a few more so Amazon EBS is pretty standard pretty common so we are going to use that type but this type is a is a is a kind of drop-down it's it's a it's a fixed list and then access key and secret key so this is I I wrote this here because it shows how you can extract user variables environment variables and stuff like that within your within within Packer and then specify the region source am i this is the base ami so you'll see a lot of this base ami term in there so base ami is ami which you start with right it's a chicken and egg problem you're just start somewhere so base ami is the one which is published at Amazon which you will take and you will customize and then instance type is t2 micro so this instance doesn't mean that you can only create t2 micros once I create an ami what it means is that the the way Packer creates an ami for you using this ami it needs to launch an instance it needs to take our instructions and then what it does is it snapshots that instance to build your ami will see that in their steps but this is just to tell Packer that hey when you launch an instance using this ami ID use 82 micro and for my usage I found t2 micro to be perfectly fine you don't need anything bigger than that ssh username so since this is this is an amazon linux it is easy to user for Ubuntu it be a one two but generally you know this is like don't worry about it it's easy to user for Amazon and I am I named so this is the custom name I gave it scale by the bay ami and then what you do is if there is an option to inspect this JSON so it catches any syntax errors or anything like that which you have it tells you what it found it found the builder as Amazon EBS and there is no provision or probies night is something we'll talk about soon so and then this is where you build it so building is the interesting part right so what it is doing is that it found the image right and then it wait it was waiting for the instance because this is where it is launching a key to micro using that instanceid oh using that ami ID so it has launched that instance now and then what it does is it it waits for SSH to be ready by the way this output is huge I've just pasted like a snippet of it but it does it is very verbose it tells you everything which it is doing and then it creates and then it it tries to SSH the reason it needs to do an SSH and red box is to run your Provisionals this definition which I did it's just like hello world for Packer there are no Provisionals there what we'll do is write it will basically it will be a copy of that amazoness am i right the only thing is that it will become private to your own account and we'll see that so waiting for the ami to be ready and then once the ami is ready it creates a security group also and that security group will allow SSH access from your account to that from your IP to that ami which came up so it can run provisioners because in order to SSH it needs the security group to be able to get into that right so you have to make sure that if you're launching all this in a private V PC you have a NAT gateway or Internet gateway connected to it so those are the kind of things you have to look at and then basically it is done and then it created this ami so what it did was it launched an instance and took a snapshot that's probably the only way I know of creating an ami and Amazon is read a queue of snapshot and instance so this what it did right and it created this ami ID and then you can see it so if you go to the AWS console you can see that a ID here so I created it it's an H VM and the name is scaled by the BMI I've removed the account ID but this where the account ID shows up okay and I'm the owner I remove that idea also and then you can see it here so when I say like I need to launch an instance that EMI shows up right and the only thing is the only thing is you have to make sure that you click here owned by me so you're not looking for this am I in a bit list right so we filter this out and you can see it here right but the thing is like we have an EMI that's it right how does that solve the immutable deployments problem we just create an ami we didn't do anything else with it right so in order to do that we need to provision stuff on it and by provision I mean we need to make it useful to us right and we can make it useful to us by installing application servers on it by doing something with it right and a standard way to configure machines or a.m. eyes or VMs or any kind of like operating system level raw virtual server is to run you know either ansible or share for puppet or we can just run shell scripts right and that's what we'll use for this for this talk but you can use all of these provisioners so Packer supports all of these provisioners and then there is more just like the providers I was surprised or the builders I was surprised to see how many exists there's something called salt 2 which I have never heard of yeah I was salt yeah so I was sorry I'd never go down into salt before running into Packer and check it out definitely so there is shell of course you know so we have to make our am i useful now how do we make it useful we apply patches we will install open JDK installed Tomcat download the war file from Nexus and then configure Tomcat to run at the startup so once I have burned all of these things in the ami and I launched that ami it launches pretty quickly why because I'm like you know I'm not bringing the ami up or bringing the instance up then running a startup script and some people do that I've done that a few years ago where you configure the startup scripts so in Amazon ec2 there is an option there is a startup script I forgot what it was the technical name for that is but there is a basically a a binary script which you can stick in there which runs out instant startup so what you do is you can run your share for you can run your shell scripts and all of that stuff at the startup but the problem there is that now you have a base ami and then you are installing Java you're installing Tomcat you are installing whatever runtime you want again I keep talking about Java but same thing for Python or anything else you are installing a runtime your application server and stuff like that and then so if you're running in an auto scaling group where how many of you use auto scaling groups on Amazon yep so when when you get a lot of traffic your ASG auto scaling group Auto scales which means it launches more instance which look exactly the same so imagine right when you are hitting a lot of traffic and you are launching an instance which is still installing Java and Tomcat and meanwhile you are getting bombarded with traffic right so it's not worth it right so by doing all of this by provisioning all of your runtime infrastructure in the ami you are saving all that time so you are your EMIs or your instances when they are launched they are ready to take traffic right away without waiting for a lot of provisioning there is nothing to provision once everything is provision and ami yeah yes we do that web console yeah definitely you can and that that's it that's a good good strategy but a lot of times what happens is that you're running behind a load balancer so essentially all your instances are behind that load balancer and that load balancers their interface to the external world but yes you can use console and there is like a post provision or scripts I'm not covering that in the stuff but there is post provision scripts where you can do that we do have lambdas but not spot instances so let's provision the ami so here I'm using shell right because I thought you know some of the audience knows puppet some of the audience nodes chef everyone know shell right so there's like that the the common ground we all agree on even though in the real world you would use something more professional than shell so this is the shell and the reason I use shell is just to give you an overview of what I'm doing here so remember like we use we had a builder snippet and this is a provision or a snippet there can be multiple Provisionals you can mix and match this is an array Provisionals is an array so you can have multiple of them you don't have to like well I started with shell now I want to do puppet yes you can so what we do is we do yum update - why obviously sudo yum install so I'm installing open JDK and Tomcat and then I'm removing Java 7 and then I'm doing a double you get now again this is like a little crude because I have a hello world project on on my github and I created an artifact for demo purposes and then this is what it is doing is that it's downloading this in the real world this would be an artifact from your Nexus and in the real world you would pass version as an environment variable or a command-line argument right so now when I'm creating an AMI and then I'm doing a GN zip so this goes in a word file and then I'm making sure Tomcat comes back up every time I launch an instance so boom I have now nothing else to do in my application right all I do is I launch the same I and I got everything I need right and this is what's going on so I change the mi name to whit Tomcat and then I have a provision ER so this is something which you had in the previous slides and we also have provisioners here right so we added that provision ER and it did the same thing except at this time it went through the whole Java Tomcat and github and everything in there and then it built this am i 73 II d5 509 we can see it there so now I have two m eyes one is without Tomcat one is with Tomcat with my application in there and then I can launch an instance with it and we can check it out so we can see here is de mi name come on come on come on this is the instance yeah there you go So mi ID is with Tomcat 8 and then I can go and I can check it out and it did bring up my town guard by the way I do not recommend this in the real world in the real world this space should not show up when youwhen you're deploying Tomcat but this is a talk so you can see that Tomcat comes up and we deployed a war file right so that war file has an API in there called hello world we are gonna hit that API right here so what I do is I do a curl on the same address slash hello world - API slash hello and I get the response hello world right so now what did we do right we did not deploy any code at Build time or anything we have any mi I can deploy as many instances of this ami and I'll keep getting this if I have a new version of hello world hello world - righto I can use that version - now you know basically do immutable deployments now there is no conflict between versions or anything else right and you automate this whole process right you basically this is what a Jenkins job would look like so you'll do all these steps except you'll do them in Jenkins like so this is the summary of the stock is right do not release code release infrastructure and this is important right you write code but release infrastructure you do not release code in automate everything and obviously you know legendry mode is when you do not need to do SSH on your instances at all right these are the resources so this Packer site ec2 and then I have a blog post which covers all of this including the Jenkins job and there is a gist and everything in there tinyurl Packer - Jenkins you can go there or you can go to Lobster on 2 3 4 . github do dude that's where I have a blog post about this and then if you have any questions if you like what you saw this is what you'll be working on at marquetta so this and much much more so any questions please let me know and then follow me on on on Twitter you can DM me as well and that's my blog questions this is all this all I had questions I can help answer which one does summary or resources yeah yeah right now we do not we have we still have SSH what we are trying to do is we are trying to minimize the times we have to SSH so we use ELQ so a lot of our logs are on ELQ and we use data dog as well as New Relic so very rarely we have to go to SSH on the boxes but hey we are using Packers so that's the good part so we are trying to see like how less do we need to SSH so far we have not had there is only a few instances and the example would be a log file which we are not ingesting in Al because that's relatively new but then does the change we do in the next AMI yeah yeah that's a good question so cloud formation again you know the way I I would think cloud formation to be much more advanced the cloud formation I would use to provision things like things beyond EMI things like auto scaling groups things like use things like SNS mass SNS topics so any kind of Amazon resources I would use cloud formation but any application or deployment related resources ideas Packers so that's how I would distinguish between cloud formation and des same with serverless right I use serval as framework over cloud formation in again because serverless framework I find it to be much more friendly than cloud formation but yeah to answer your question definitely cloud formation I would use for Amazon resources and for that by the way we use styrofoam and that is also from Hoshi curve so yeah about a year you know it is it is really good really good yeah yeah it only cleared semis so launch configurations you have cloud formation to do that or terraform vo styrofoam for launch conflicts yeah yeah so that so there is like multi-step so is there is snapshot and then that snapshot bit is to be registered as an ami so those steps are there so if you were to get rid of this then you can go and you can go to the MIS page and D register this ami and it will no longer be available yeah that's a that's a good question I I did not find anything less yeah core OS is the bad minimum right yeah yeah we haven't looked into core OS yet but definitely with like I take a look at it because that that is really good good point where you want to start with bare minimum and build on top of it versus start with something which is a little little fatty and then try to find out what I can get rid of like I don't need Ruby for anything so let me get rid our Ruby but like I'm doing a lot of guesswork and again you know yeah exactly if you share then you are tied to Ruby because if you are using a chef provisioner so stuff like that lot of guesswork a lot of human intuition and stuff comes in any other questions folks you do have a storage cost for the mi but because that EBS snapshot you are paying by the hour for the for the ec2 storage or EBS storage and then you will pay the cost of the instance of time when you launch the instance but when you're running Packer the instance comes up it gets snapshot it and the instance is shut down so you're only paying for the snapshot storage Atlas is also a Chicago yes yep you can use that repulses like a repository I think so yeah thank you so much yeah thank you everyone great great talking to you guys thank you