SBTB 2015: Vidhya Narayanan & Chris Webster, Language Choice for Scalable Platform

Recording: SBTB 2015: Vidhya Narayanan & Chris Webster, Language Choice for Scalable Platform

so as aleksei said I work for Verizon I'll invite chris webster to the stage in a little bit but the way this keynote came about is I we ended up submitting a proposal for a regular session and then Alex I reached out and said it's a great topic you want to do a keynote I think he felt that the session was not technical enough so he put us here but so here we are the topic was about kind of scala and using scala as a means to build you know a reasonably sizable distributed system a platform and the time we started on cue a few years ago you know when i joined them we were a really small teams so there was a lot of pressure to build both the product and the team from scratch so often wonder when I kind of step back and think about it if the choice of the language speaking Scala had anything to do with our time to market right so that's kind of where this thought came is to kind of share some of the experiences and challenges that be kind of surfaced along the way so on cue as Alex I said was started at Intel and was bought by Verizon last year and we are building an over-the-top video platform so when we my team is responsible for all the backend a server platform that's hosted in the cloud so when we were looking at technology choices and to be fair some of the management team that had started this project where ex-microsoft folks so they actually felt very liberated to be able to pick any technology to build this product from scratch right so it's fascinating if you come from certain companies well that is mandated to use some very specific company wide technology it's amazing to come out there and look at all the options so when we were looking at the various choices Scala kind of stood out for many different reasons I just the principles around the functional programming ability to write concise code kind of the principles on immutability and because we had to build a scalable platform kind of the design patterns around asynchronous programming all of that was appealing and the fact that it's JVM based language was comforting i worked at oracle in the java group for several years so the JVM option is always something that feels comforting so scala was a natural choice kind of in that regard just just to kind of give you a sense for the size of the product that we are developing we have about close to 200 services out there and these services are essentially you know many of them are a client facing so any kind of content management user management kind of personalization of that any kind of data that any of our client needs so are served by many of the services then on top of that since I talked about the video platform we have recommendation engine all the advertisement capabilities for dynamic ad insertions or a detection and all the data you know delivery pipes both the pipes that take in the data and do the analytics and the reporting everything is color code we have lots and lots of Scala code in our product and it's a pretty reasonably sized developer team as well at on cue and we are kind of rapidly doing development so we kind of exercise our muscle to go as fast as we can so we do several deployments in the production environment as well it's when I said a video platform you can just imagine any other application that you're used to right so the type of products that you'll be seeing coming to the market would range from our own build first party set up boxes to mobile devices to apps that runs in other consumer electronic platforms as well so it's has you know an opportunity and verizon wireless's has a huge customer base so kind of the mobile offering the expectation is we have a lot of people I'm sure a lot of you have verizon subscriptions so hopefully use our product there so expectation is there will be a lot of usage and with that then the constraints on the platform to make sure it's fast it's available it's reliable so all those qualities kind of come into the picture so what we'll do right now is to kind of walk you through some of that so let me invite chris webster to the stage chris is an Associate Fellow of it verizon and he manages the team that does the client facing services hello Chris everybody and so we will hopefully do it in a little bit of a fun way we will do it in Mythbusters style so what we wanted to do was probably throw some popular myths out there or make up our own and we will talk about our experience and then we'll say whether we prove our busco Smith's does it work all right you want to kick us off with the first outrageous one hopefully out there sure so I think this crowd is already going to think these are all busted anyways but that's okay we're going to go through the motion so at video as you mentioned we have lots of developers writing code and you know we have multiple web services they all have to scale we have lots of use is skala really Enterprise ready for the enterprise not enterprise-ready but ready to be used within a big development organization what do you think all right so I'm sure many of you get asked that question if you are with big companies why Scala why do you have to do this we hear this all the time especially as a manager managing a team doing Scala you often have to put that you got to go batting for your team it's not easy to you know stand out and explain which of the next version of the refactoring you're going through or some issues on some library you're trying to adopt has to do with Scala as a programming languages as opposed to just the fact that we are building a scalable platform right so one of the things that naturally occurs when you adopt any kind of open source technology is you're often looking for libraries or frameworks that help you get up get things done right so it's @j either you're looking for a json parser you're looking for suitable web frameworks in our case or any kind of logging frameworks or libraries we are often looking for options and it's not something that's unique just first scala even in the mature language i'm sure most of you have to kind of go through that exercise at some time or another for the what we observed though in the case of scala since it was new at the time we started adopting the ecosystem was still maturing and there was kind of rapid consolidation happening in several of those libraries right good examples where code coverage tools we are struggling to find good code cartoons and you know the the plethora of JSON parcels that are out there so the the size of the team doesn't lend itself to go off and look for libraries especially when we try to solve common problems across our platform right so one good strategy that we have adopted that seems to work well for us is to have a very core team go and look for options to solve some of the very common problems across this large system right so concrete example I'll give you is when we migrated off of third-party hosted cloud database to our own Cassandra clusters we had to think about the database driver that was the first problem we ended up adopting a library and we soon got into an issue where it was it didn't have the community support that's needed remember the Hector train that we went through and then we ended up adopting you know datastax a success and our company they're their own drivers and that worked well for us and Dean was there with us trying to help witness the journey of all the madness we went through with it so we had a interesting data access layer that we were using across all our services to talk to the third party hosted system so the first effort we took us have this core team build this new data access layer against our cassandra clusters so outside of hunting for the libraries the core team had to understand the various use cases of how the database was used and we had to make sure that be prototyped and understood the pros and cons of you know what what we were trying to build once we got that then the next problem is understanding what is the performance overhead of this new library so we took some of the industry benchmarks we ran them against the cassandra cluster had a good baseline and then we made sure that we were not adding too much overhead by our own library and then we had to compare against our older data access layer on this other database we were using and then once he got to a good point he said the rest of the team was able to just go and make a change and all this time the folks on the other who are working on the other systems were thinking about how they were going to migrate their business logic to the cassandra database thinking about data model and the proper way to design it as opposed to worrying about these things we ended up following that same model when we go through you know any kind of big changes every scholar upgrade and play upgrade is takes a long time because of just the amount of systems and code base that we have so we kind of end up adopting that as a strategy in a distributed system again failures are inevitable so some of the design patterns around dealing with that I believe is completely agnostic to the programming language of choice right the systems have to deal with failing gracefully so in our one of the common library we have where we have a service to service library that does both the discovery of the end point you want to talk to and also does the communication across these services we have some patterns that be a bill for circuit breakers right so if the end point you're trying to reach to reach out to is slow then this nose kind of how to deal with you no timeouts or whatnot and the fact that the other endpoint is not reachable anymore means we have a backup strategy to make sure we are not D dancing our own systems it's the D does problem is all about our own bad code doing all bad stuff so there we kind of put that design pattern and we deal with it in some of the common libraries so that I believe has nothing to do with Scala as a programming language um but there are some patterns in Scala that helps with thinking through failures right so the scholar composition model it's a good way to model failures they again making sure that we are always keeping immutability in our mind and write the code means we are not dealing with some of the nasty bugs out there and then the non-correlated stuff right when it comes to denote resilient building a resilient platform and that has to do with the kind of the operational side of things right so when you think about the operational side we are big users of Netflix's simian army so we have a be induced both latency and failures in our production system and we make sure we test them for how we expect that the systems to function in case of failure so test test test that really helps and also you know kind of needless to say a lot of the people do this which is go with you know kind of redundancy as a strategy to deal with failures rights so making sure the stack runs on multiple data centers and then you deal with the problem of how do you keep the data in sync how do you you know make sure that the requests that are coming into a data center in the case of failure fails gracefully to the other available ones so getting that all right has nothing to do with skull as a programming language and the last one you know some of the basic things let me just get them out of the way right so having a continuous build environment or integration environment make sure you have end-to-end test that tests the common scenarios like common sensical things and make sure you're running them on every build so you can catch regressions ahead in your development cycle as early as possible really helps to address all that so those are some of the basics outside of this something that I'm passionate about and it's work bill for us is really encouraging the dev team to do deployment as often as possible right so the act of taking your working code from your laptop and dropping it with the rest of the components in a distributed stack is is really important there are a lot of things that we find by just going through that exercise rights the performance characteristics is very different or just how the interaction model works in the distributed system means there are lots of things we find that we don't normally find when you're just building things in your in your local environment so that habit of going through and exercising the deployment muscle really helps so some of the examples we have a fairly good automated pipe that takes our build from a you know less restrictive to increasingly restrictive environment if I have to put it that way and so we find limitations on that when we build a new system so you know Andrew talked about having people doing multiple things we try to do that too but as you kind of scale the team you do have you do get into some lockstep are bottlenecks right so the sooner you find a dependency that you're waiting for someone else to change their library or code the faster you will have a path of taking your stuff into production right so that becomes kind of unavoidable in our case as much as we try and have people jump on various things and work on there are some areas that really becomes hard for everybody to jump on and do things so you know kind of that kind of summarizes most of what i wanted to say on this topic what is your verdict what do you think Chris I think it's busted I think we've been using this for a long time as video Mitch and it sounds like it's busted to me what about you guys last day good okay veal busted all right you want to kick us of the next mint yeah so the next myth well there's a lot of Skeletor but the myth is it's hard to find people who can write Scala it's going to be impossible there's only you know a few people of people over the world you're going to have to get them you're gonna have to fly all over the place and get him what do you think Vidya hey um when we started ramping up the team and Chris knows this ville he was one of the or very early hires in engineering group we were so worried about the ramp up time needed to learn a new language we were hell-bent on finding people who have skull has skills got very very difficult I'm talking about you know 2012 things have changed a lot just in the past few months or so but that's it you know it's really important to build a team that kind of works well together we can have super sharp people with very good language skills but if they have an attitude that kind of ruins the experience for everybody else so we used to have comments like oh we want for textures we wants 10 mixers we want people who walk on the water and we still kind of do that but the fundamentals we look for a vanilla for people is are they good people to work with it sits at the end of the day it's the team right and do they have good foundational aspects that they have a good aptitude to learn right they have good algorithm background are they familiar with distributed systems have they done high scale do they understand the need to do performance optimization so those are kind of fundamentals that we look for in the team right now and so you know on one side you have these really good language enthusiasts who are very adamant about making sure that I'd idioms are used in the right use cases on the other side you have these people with lots and lots of experience doing things for real and they kind of have know a lot more about you know a GC or internals of the JVM and how do you travel shoot and how do you profile your system so that kind of makes this really good and it's been working very well for us so that's kind of where we have shifted our hiding strategy at this point I have to say and then having big Scala as something that is new for us we I don't think the community took a seriously right whether it's verizon or internets kind of a corporate thing so starting to feel relevant in the community needed means that we needed to get some of these key hires on board one one strategy that worked for our very first hire that we had is our X VP flew out the UK had a breakfast with this guy and that was enough to convince him to come and join us and and once we kind of applied some of these strategies and had some core folks who are relevant in the community here then it started to grow kind of organically right there are lots of our so you know folks who be kind of brought in that way who are there doing technical sessions art scholar by the bay or escala de Servian we host the meetups and we have open sourced several of the scholar libraries that we have been developing in house as well so I see still there we have runar we have Tim you know you probably have come across those guys giving talks of several scholar conferences and next time you attend Tim stock ask him how the breakfast was and you know though a lot of the developers are exposed to multiple languages it the learning curve does differ for different people there are people who come in and day two they're ready checking their code built the functionality they feel like they got it and there are people who take a little bit longer to kind of get a feel for the the new patterns of doing things and it takes while so but as you know as someone on the management side something that is really important is to make sure you are making all the venules and everything available for the dev team to feel like they can learn and conquer this new thing that we are marching towards right so we do a lot of scholar trainings we had a type-safe consultants it right close to us you remember that and so nothing nothing like writing a piece of code and I remember you know people go around and sit with this expert and like can you look at my code can you review what did I do wrong not wrong so that iteration helped a lot in kind of getting out of that resistance mode and that that worked extremely well for us so you know meet ups and we do cross pollination across the different groups in verizon we are not the only group that is a big group in in South Bay also doing machine learning their big heavy users of spark and so we make sure that the teams have an opportunity to cross pollinate and learn from each other I kind of gave us out I believe this one is busted as well I think it's busted what about you guys you don't have a choice other than agreeing that that's it listen to us all right Chris you're gonna do the next couple yep I'll do all the do the next few i'll be I need a cheat sheet though all right so this one is is another interesting one right so that's speaking a new language or picking Scala do you think it has had any influence on the culture and since I have to freeze it like a myth you think it had a positive influence on our culture so when when i first started was pretty pretty recently after Vidya joined and we had to mix a few people that knew scholar really well and a lot of people that were coming from either Java or we had ruby on rails or even had some C and C++ developers who are on board and everyone was kind of learning together so Park part of the part of the the interesting thing about learning together is that people aren't afraid to ask what might seem like a dumb question when you've been doing you know development for 15-20 years and you ask Oh what does double equals mean in Scotland for coming from Java right it's a little bit different there so there are these things that are a little bit different and since everyone was kind of learning together we had recent grads we had lots of experience people everyone was kind of working together a lot more than they would if they're experts in the domain they're just going off being in our code and everyone's kind of remote and siloed and everyone's doing the job but they're not kind of cross pollinating and so since I think we started a brand new kind of greenfield project and a lot of people were learning the language for the first time it was really kind of empowering to have everybody working together at a level that they probably wouldn't if we started with you know if everyone's coming into an existing project and we had already had experts they plan on what it just got off and did their their pieces the other interesting thing about our you know scala in general is that it seemed like the experts that we haven't even the new to become experts we're not just interested in learning but also in teaching like Stu he's sitting over there i'm going to pick on him since he's here i don't want to talk about in his back it is really good he has a brown bag session we have another person on our team that has a weekly tech talk and it was usually just scala topics but he's expanded it out to do other things and it seems like that has been just a lot more prevalent I think one is that people are interested in learning from people who have more experience in different areas and I think the you know that trick create a new language where there wasn't a lot of expertise in our team where there was pockets of expertise with one of those things that kind of drove that the other thing is nvidia kind of alluded to is that we had you know a lot of people that weren't scala experts that but that did have a lot of experience in java and performance profiling and cassandra and those kind of things and so when you kind of bring these people together really everybody has stuff to learn and everybody has stuff to teach so everyone really feels like you know somebody can can teach and you know you see me talking up here i have to do some of the sessions to they force me to do them that is the teaching part of it not the the learning part i go because i like to the teaching part sometimes is a little little little harder but you have having this concept of everybody has some to learn everybody has something to teach really is kind of permeated at least my team and i think more of the environment that we have and just a couple of other examples that we have from the team is that we had one of our data scientist who had done a lot of matlab and our programming was seen you know everyone's picking up scala and you know she started to do it and and became a good scallop programmer we had a very experienced gala developer I kind of mentioned this earlier who's running these technical sessions these bi-weekly technical sessions and kind of take it upon himself to to get interesting sessions and if he doesn't get one he does it himself so it's that's always good and one of the interesting thing about about stews session is that there's 11 engineer on the team who is very diligent and does every single homework all the time and she's only one who's been able to do that and intends all of them so really the driver in this case for for the positive culture was definitely in the new language but it doesn't have to be in your name which it could be around some other new technology or something that kind of everybody gets to learn together the reason I think skull o is very important is that it seems like the more you learn about skull and functional programming in general the more you know you don't know so I'm at the point where I know a lot but when you know there's always a lot more stuff that I don't know and when I talked to stew and some of the other guys on our team I realize hey there's really a lot that I don't know and so I think it's those things that really drive our culture around here what do you think Vidya well I think you made the point that a lot of the behaviors you see it's just not scala specific right and i think sometimes we forget what it takes to learn a new thing if you've been working on let's say java for a long time and you get comfortable about it there's no empathy for people coming in and learning something new I think that's fact right it's just like you don't get this that's the kind of feeling that that comes across so I think you know everybody isn't the same I have a leveling field and everybody is learning so I believe this is possible I think there is something to say about how our culture has turned into a very collaborative environment thanks to picking something new so let's declare this possible anybody have a second second okay we're gonna go on anyway so all right the last one all right this hard one call us hard I think it's hard so there's this an interesting question as I mentioned before when we started adopting Skala we had a lot of people from you know a lot of Java programmers some C programmers and you know functional programming is certainly different right so it's I think part of it is that people in general are resistant to change it's not just about your programming language it's about try to take take away somebody's editor tell them they can't use VI anymore tell them they can't use Emacs or whatever their favorite idea is and you'll probably have a resignation within a few hours so you know you're talking about moving shifting a programming language which people feel pretty religiously about and some in a lot of cases I think so a change from from coming from Java you know you're not just talking about the language you're also talking about tool changes of in Java if you are using aunt or maven you're probably now going to be using SBT which is similar in some sense and the dependency management but it has its own flavor of how you have to do things you know the programming language the API is it you have to learn the the idioms patterns you know we had lots of mentions about scholar Z and some of these other functional programming topics there's a lot of new stuff to learn and you know it's kind of scary to say oh I was an expert in this and now i'm going to go to i'm not going to know you know a recent college grads going to know more than I do about this and that's true and but you have to you know think about is it just the scared of learning something new is it just being scared about learning something new or is there more really to the language is hard it's impossible to learn I don't think it is obviously there's a lot of people here and that the adoption is increasing so that really I don't think that can be true I think the other part that's interesting is the expressiveness that Scala so usually you can write less code to do you know about the same behavior or more but part of this you this expressiveness as you get a lot of you know you can create dsl's and fluent programming and if you're not familiar with the dsl and the language that kind of makes it tough like the first time you you look at something like Scala test respects to and you're you're reading this it's like okay why are these words in the middle I time used to having the dot and a parenthesis here and it just doesn't make any sense to me but after you learn it you say okay this is a domain I understand that you don't need to use the dot in this case in that case and again it's an adjustment once you've done it once then writing those tests the second time or that maybe not the second time the after a few times it becomes a lot easier so it's really the skidding over the hurdle of being scared like just like public speaking nobody wants to do it but once you do it 15 times you're a lot more comfortable up in front the other thing is there are some interesting things if you come from Java there's some you know it's kind of like Spanish and Portuguese there are about fifty percent similar but the fifty percent different to usually bites you so for example Java is just fine returning Scala's there are some problems with that so you don't generally want to like there's there's a double equal in Java look at it you don't you just don't do that but in Scala we do that there's the you know just a few things that we get burned on the first few times it's it's it's hard because you don't know or forgetting to put a if you don't declare the return type of method and forgetting to put that equals the very first time and wondering why your code doesn't compile it's returning it you know why does that work I don't know but you learn it you learn the hard way once hopefully the compiler and tools help you but that first time you hit it it's just hard and it's hard because it's different the other thing I like Vidya mentioned using a kind of core set of libraries where you have a lot of examples some of the stats up there we have hundreds of services so you know if each service kind of took its own set of libraries it's really hard to jump across three different services that know it know what they're doing even if you're an expert in the language if all the domains and libraries that were using are different so we kind of tried to standardize on at least some of the library so you can move across services and look at the code and at least read it a lot easier than if everyone was using their own libraries and idioms and finally I think the last thing that we kind of adopted is you don't necessarily need to use every single language feature just because it's there there are cases where you do but you don't have to do that and so using judgment that that problem never really goes away it kind of stays there the last thing that I'd like to say about whether skele since we work in we're creating a lot of web services we get a lot of high scale systems it's very nice that immutability is a lot easier to do than mutability if you don't do anything and use a list you get it get it immutable list from the services side the worst bugs that we get our memory leaks nobody wants to get a memory leak even a small one you might catch it in testing but some of these ones you know a couple bites here and there after billions of calls you get the dreaded out of memory and then your whole system is crashing you're rolling restarting trying to figure out what the bug is and get everything working so that for the service side if you really treat you know a web service API as if you think about that as a function most of the stuff should be immutable yes you want to do some caching for availability and cetera but if you minimize that you you will reduce the number of calls you get middle night or the weekends or the times when you don't want to get those calls so for me that's that's making it easy to do the right thing is you know the language kind of guiding you down the right path and so I think I think it's busted I don't I think that there are certainly more things that everyone can learn but I don't think Scala as a language if you apply yourself to it as any harder than any other programming language look at if you even considering to Java who's read the programming puzzlers there's multiple books right it's not just one there's multiple books so you know you can pick a language and pick the puzzle or C++ effective C++ right more effective C++ more and more effective see possible so that i think it's it's not just scala its programming in general and learning the concepts what do you think video right and we already know you guys think that you're here so that's all we thought we could chair this morning thank you for the opportunity appreciate it