SBTB 2014, Tihomir Bajic: Boosting Enterprise Software Innovation with Scala
Recording: SBTB 2014, Tihomir Bajic: Boosting Enterprise Software Innovation with Scala
all right guys so i'm here to talk about um how great it is to do runtime type evaluations um no nobody all right that was supposed to be a joke but i'm here to talk about why it's uh why actually we've adopted type save stack at nitro play aca and scala of course and rather than go through specific code examples what i'm hoping here to do is to give you guys actually a framework for adopting new technologies at large but also actually for adopting uh specifically place kalanaka and i'm going to try to cover that by having giving you uh 25 concrete examples for how we've done that at nitro uh this is actually a work in progress so i'd love to get your feedback on it and if you guys are adopting any of these technologies at your own company or your own open source project let me know i'd love to help out and share the lessons learned this has also been inspired by having been to one of these conferences over over a decade ago and coming back to work and then trying to actually go and apply what i learned in the conference very next day and talking to a number of you over their booth right by the front door over the last two days what i've heard is that you know some of you are already using scala uh your company but you know it might be still on the per or on the outskirts of the main project and some of you are really intrigued and you love scala um that's why you're here and if you don't love it you probably have been convinced about that by now and you really want to go back to work on monday and start using it and start figuring it out but what's going to happen is you're going to be faced with a number of questions so my goal here hopefully is to share a number of lessons learned give you maybe some clips or give you some ideas of how to deal with those those questions answer those and become really successful i want to frame this just by giving you a really quick summary of what we do over at nitro because it's kind of important for some of the technologies that we use severe document sharing and collaboration service uh we enabled this for millions of our business customers uh and we handle millions and millions of documents on a monthly basis uh we work with documents online or offline and we try to say that we have a global reach anytime and anywhere we started on the desktop as a c plus plus application and then last year we actually relaunched a number of that ip as a stack of sas services running in the cloud using using play and and and alka and scala and we're we're really looking to scale this we're trying to be this complete solution um working with with your devices wherever you might be so once again why this is important for us and for this talk is because we started really as one type of a stack siposplus and desktop.net services in a public cloud architecture we're already dealing with five million active monthly users over uh cash flow a positive company so why change right like why would you ever want to change well as we kept scaling we started encountering some really weird uh issues with our performance with our uptime just how expensive it was to run some of our cloud infrastructure whereas right now where we are today all of our backend services are our typesafe stack play scala aka we've done an amazing job at reducing uh the average response time and already we've transitioned over two million active users on a monthly basis uh being being the first users so it's been a journey it hasn't been a straightforward path at all uh we've had to take some detours and we've you know taken some pit stops uh to reassess where we were and what's actually happened over the last three years is like yep you know early on uh looking at playing scala i wanted to apply it immediately at my company and then we realized that we were running into a number of hurdles and had to answer a number of business questions so first actually when we play in java completely hosted on heroku then we started bringing that kind of in-house and went with the hybrid model and then we finally earlier this year went with play 2 scala aka all done and deployed inside of arcola and why that's interesting is because this is where all the you know there was a talk about the devops and how important that is and when you're doing re-platforming so so frequently and so often uh you really want your devops guys to be uh to be your best friend so i'm going to talk about for the next 10 minutes about the framework for embracing our tech stack evolution i'm going to break it down into five high-level topics and give you five uh hopefully simple takeaways for each one of these so it's a five by five structure we're going to talk about people performance product process and productivity so first thing first is you know you go back and you maybe want to apply apache spark maybe you know stitch is going to become open source you want to try to get your hands on that and you know a tendency at least my tendency was that you know go back into a dark corner and start working on it right away uh but guess what don't do that let your boss know because unless he knows what you're working on it's really going to be hard to get the the sponsors corporate sponsorship you know make sure that you champion this solution uh together make sure that you're actually a hands-on champion as well uh don't be paying a lip service as an example and and be right there speaking up at events like this or or in front of your company you know the first version of nitro cloud that we deployed it was literally an all hands meeting and it was a deployment to heroku where we staged a simple bug uh nobody knew that was staged and then we fixed it right right away just to show how powerful working with play was how quickly we could test it how quickly we could redeploy it and how quickly would go then backlab into the hands of customers make sure that you seek out early developers at your uh your own company your own project and probably the first stop should be after you talk to your boss and you explain basically what you want to do and he gives you kind of the skeptical look because you still don't have all the answers the first next stop should be the devops team and yesterday we had a really great talk about devops and and some of the tools uh uh that will help you like jenkins or it could be continuous integration continuous deployment organize uh informal hack sessions one of the things that we've done really successfully at nitro is dealing with a real production issue as long as it's not a burning issue that has to fix asap uh and and and try to get your hands uh uh try try to get your hands on the brand with the brand new technology and apply that to a hopefully they'll isolate the problem uh this is an example of something that we're going to be open sourcing soon but we are dealing with um hundreds of millions of uh s3 blobs and in our in our buckets for our customers and we want to do continue performing health checks on those and copying them and do all of those things and actually enumerators in 30s were really really great uh first step for us adopting scholar nitra make sure you also adopt a line with the tech community right there there are different tech communities out there you know scala ruby python all of those are potentially really really great frameworks or languages make sure that the one that you are picking fits you and what i mean by that is you know conferences like this are really great to actually get to meet the authors or the people behind the technologies that you want to use reach out to them and figure out you know how they can help you figure out you know they're humans they're they really love preaching and teaching what they are working on and so figure out if you're going to get the type of support that you need to get because you're starting from scratch and i was really positively overwhelmed early on how great the play framework community was reaching out to the guys at linkedin getting their feedback james wore the heroku was was really awesome so that was one of the big reasons why we actually when we play in the in the first place um and then if you're adopting a brand new technology uh it's likely that you're not the expert right because you're coming up after this conference you're all like you know open eye and you wanna you wanna figure it out um so you need to attract some best some some external talent to help you out with this you know it might be from the community might be from consulting but hopefully you're gonna be able to bring aboard full-time members to help you out whatever you do make sure you're being authentic make sure that you're being yourself hackathons are a great way to cost effective ways to go to attract new talent some of our best engineers came from hackathons so whatever you do be authentic uh early on we launched this site that's still up it's been three years up now it's called uh no site it's one of our it's one of our values at nitro and so that's really been great at adopting uh new technology and attracting your talent because that was the first site actually that went live uh using the player framework uh so let's talk about performance so you want to make sure that you're instrumenting your benchmarking your existing application or your existing production production solution so apm solutions like new relic app dynamics you should really be investing into that first and foremost before you start moving and ripping things apart and changing them because this is how you're going to actually in the end going to be able to prove that what you're doing is somehow better because if it's not better why do it or why adopt it globally across the company this is a mouthful but i hope it makes sense you can only optimize what you measure and what you measure is why you optimized so beware of uh vane metrics so at nitro we had these like super awesome expensive big tvs that were they were showing us live stats about their users and then when i launched that stuff it was awesome i was personally overwhelmed because first day we launched nitro cloud we had something like hundred thousand people use it and it was really awesome just watching you know numbers or emails or or names of people swinging bar on the screen but guess what that can be really distracting if that is not what you're trying to optimize for right we didn't have an issue with the top of the funnel we had an issue with document processing and once we realized that we needed to you know move away all the other things that were distracting to us and and focus on document processing performance it was not until then that we actually started using aka for real we had backup prior to that for a whole year but we were really not optimizing it um measure and optimize your deployment speed so uh talking to some of the guys yesterday uh some of us are on on bi-weekly cycles some of the guys are on a quarterly release cycle and some guys uh release the production several times per day and you know how quickly you can release product even if you're doing it once per quarter is actually really really important so make sure that you optimize and monitor and instrument all of that i know there were some famous articles out there several years ago talking about sbt performance building scala and all that but for us it's been amazing sbt module code jenkins nexus we had a presentation about that yesterday actually works really well measure and minimize local dev time or or dev idle time really and what i mean by this is make sure that you track a few things that are really really important to you when it comes to just how performant your engineers or developers are so in nitro we used to not not anymore because it's a bit cruel the entire engineering team wouldn't go for lunch on the first day until the person was completely set up and was able to push their code to production and then we got 20 people and then somebody was always hungry but make sure that that you know you optimize for things that are really important to you and make sure that you again work with devops because using frameworks like play you can make the local redeploy time almost unnoticeable last bit about performance is someone's going to ask you to prove it someone's going to ask you to either prove why it's better or they cost less or that you know it's not going to break the bank and so build roi calculators really early early on because someone's going to come asking for it and for us it was it was important to really talk about the cost of s3 because again we ship you know millions of documents around every day um so speaking about product i'm going to go through these slides the process slides a bit a bit quicker um because really you need to treat the main message here is you need to treat your tech stack as a product so you need to uh make sure that you're not adopting new technology just for kicks you know that's research that's not development um so so so make sure that the business understands and make sure that you as a as an engineer are being pragmatic and and you're really shipping code fast and meeting business needs and you're going to get your chances down the road to really uh maybe play with some cooler tools but you have to earn your chances one one of the things that you want to do as well is when you're adopting a new technology is pick a high impact functional parity project to get started and by this i don't mean something that's absolutely core uh to your company but something that hopefully is high impact because it's not performing right now um and adopt the new technology to to work on it because you're in you're in training wheels mode early on and you're you're trying to prove yourself uh don't ignore the table stake features like performance or or uptime or stability because while it's really cool to you know maybe maybe adopting just naming it you know apache spark unless unless your users are having logging problems like potentially they were having at work that you don't get to do some other cool technologies until you handle this otherwise you're not going to have a job a month afterwards optimize for for the speed of innovation and and here what i mean by that is really how quickly you can release and really measuring uh measuring how many uh tech stack evolutions you can have um and of course you know treat the tech stock as a as a evolving product itself meaning that you actually need to apply you know proven product management practices to it like iteration uh quick iteration quick releases measuring metrics and then and working off of that um the you know one liner here that's really the most important one about the process is that devops solutions matter uh chef plus vagrant we're really really huge uh fans of this at nitra we're going to be hopefully open sourcing some of the things for us and when you're working on an existing product code base and you're building a new stuff you have to be able to switch frequently back and forth chef plus fragrant is is thumbs up democratized research and solution design so you know you come back from a conference like this and you think you want to you know learn and apply all the lessons learned yourself uh don't do that you know talk to other senior members across the team and build a bit of a ground swell together focus on demonstrable results first and foremost as i said uh when we were trying to start as a first engineer at nitro in san francisco my first job was to actually create a recruiting site and i used play framework for it and it's still live right now at the slash no side so quick results fast except that you will need to iterate this means that even even if you are applying a brand new technology understand that there's going to be lots of quirks to work out down the road another really important point when it comes to process is we heard a lot of companies a lot of people stand up here and talk about how their companies apply reactive engineering principles if you're doing that make sure that you apply that to your stack as well productivity you know engineers are really expensive as resources you want to make sure that you reduce non-engineering work for your engineers early on before we had even the devops team or we could we could have four resources like that we're all pushing directly just to get push deployments to heroku uh be careful why you measure when it comes to productivity a number of lines of code comparing number of commits might actually gamify uh things the wrong way and you might actually be gamifying blindly so please please try to avoid doing that you know for example for the for the sake of fun had a jenkins drinking game type of thing going on in nitra and that created some disincentives where people uh really felt potentially uh targeted because they were not committing as fast as other people and that that's just the wrong way of looking at the metrics that don't matter at all and the point that i had in the previous slide was uh you can actually apply there there are metrics that matter and those are the metrics around the code the test coverage how frequently your code churns and produces issues and you can do paper trail nagios a whole bunch of monitoring solutions to really try to figure out if your new technology that you're using is helping you for example eliminate stupid mistakes like one off or off by one mistakes or not pointer blenders the last thing and this is the last slide i want to talk about is make sure that when it comes to productivity you're actually talking to your fellow engineers productivity really is a mindset for example i love the play framework from the moment i i deplored my first half to production with it but really productivity is a mindset and there's a number of different tools that you can use to become productive i'm going to be making the these slides available you can ping me on twitter you can write me an email or you can see me at our booth i'd love to help you guys figure out how you can apply or adopt aka scala play your own company and take it from a guy who had to sell it across the company several times thank you very much yep sorry the question was about no stack in production or i couldn't make them here oh right so how do you release the new stack in production uh you cross your fingers and pray nothing or goes badly um that's where having a really great devops that cops team comes in place and we were just yesterday talking about this you know there are you know a number of solutions when it comes to working with your cdn like cloud for rollout roll back redirect some parts of your traffic gatling was was on display earlier today when it comes to performance testing locally before you go to production and again that's that's probably uh a whole conference just dedicated to that how you simulate real load where we have like five to ten documents being uploaded in a given second um so that's a really good question and it took us months to resolve that one branches like continuously like data migration like my own system is taking your data and how do i continue like this data system those kind of things so data migration as well is that is the question right um so this this is where it's really important when you're for example applying a new technology they try to isolate the number of moving parts right so you know that was a silly example really where i stood up a a static marketing site using play framework uh but that was a great example from the technology standpoint because it had very little interaction with the existing stack right so if you are for example changing you know you know we had an example of authentication performance issues so so work they from what i got was not really changing the underlying data store or or changing really the services they were changing the middle layer the orchestration of it so again going back to my slide about apm where you want to instrument your application with new relic or or app dynamics or whatever it is understanding where the bottleneck is and then take away all the other stuff keep that fixed and just focus on that so it might be data migration issues but usually it's not like usually there's other issues that you need to solve first it's not like throw away your relational data bill replace everything using mongodb that is usually not the solution all right thank you guys you