scala.bythebay.io: Scaling Scala Teams Panel
Recording: scala.bythebay.io: Scaling Scala Teams Panel
Then let's go straight to introductions. First I'll say I'm Alex Krabarov, I'm the founder of this conference and I run subscala and seven different other meetups. Basically my main concern, I want engineers to be happy, so I'm explicitly on the employee side. It's kind of funny, I help actually connect engineers with companies and when I do this I say I'm going to interview the CEO and the CTO. I've been an engineer, I've been data scientist, I know basically what people like me appreciate and we appreciate engineering culture. I want to make sure that basically that's the case in the company which wants to come to our community and wants to hire people from this. That's kind of where I stand on this. Obviously as companies grow, folks and engineers find themselves in the position of power and then they have other criteria
So I think it's a very interesting question, we're kind of all living in this workplace world. So that's kind of my take on this. Hello. Okay, hi everyone. I'm Iti, I'm a tech lead at Twitter. At the moment I've been here for five years. I've been an engineer, a tech lead, I was a manager at one point too. So I've gone back to being a tech lead
So I've worked in teams of several sizes. I've been a tech lead of people, three people team. I managed 18 engineers at one point. I've been a part of a 100 people org. And obviously part of the whole engineering group, which at the moment I think Twitter is about more than 2,000 engineers. So I'd like to draw from that experience to speak to how to scale Scala teams and how we've done it at Twitter. So prior to Twitter I was primarily a Java engineer. And so all my experience in Scala has been at Twitter
So I'd like to draw from that to talk about how we went about scaling teams within Twitter. And what were the different attributes that we focused on in order to be successful at what we do. Yeah. Thank you. Hi, my name is Tiho. I'm VP Engineering at the Long-Term Stock Exchange. This is my fourth startup joining as a founding engineering member. So I like maybe working at Twitter at such a large scale in terms of number of people
Always been working in the teams of, you know, a few couple to a few dozen to maybe a couple of hundred. And I really focus on early on the whole experimentation moving fast, learning something about the customer and the opportunity and how they use technology to the best advantage for that. And had an opportunity in the past in my previous startup to move the stack from into Scala and in the new startup as well. We use a bunch of Scala backend services. Hi, guys. I'm Vitaly Gorin. I'm VP of Data Science and Engineering at Salesforce Einstein. It's the Artificial Intelligence Division of Salesforce, home to companies like Metamind Prediction.io and some others that we acquired along the way, including our own internal organic researchers, engineers, data scientists
My first kind of encounter with Scala was actually a fairly funny one. When I was working for a company called LivePerson, we acquired the company called Amides. And when we actually went to implement their technology and integrate, we kind of found out that it will be faster to rewrite it from scratch and actually try to integrate. I said, well, you know, if we already rewriting it, then why not use the new technology? We were a Java stacker at that point. And we actually rewrote it completely in Scala, which was about kind of five or six years ago, which was fun. And for me, kind of the way I got to it. And then when I joined, I was kind of one of the founding members of the group that is now called Salesforce Einstein. So basically, when I joined the company, I wanted no other technology to use
And now we're also probably the largest group within Salesforce that uses Scala. Hello, everyone. My name's Tim. I am the infrastructure engineering lead at Verizon Labs. So I can't fix your phone or anything like that before anyone asks. So, yeah, primarily we work on infrastructure. So what that means is we basically provide all the libraries and sort of architecture choices for a fairly large service platform. And I've been doing Scala for about ten years
So, yeah, if you joined in the last probably four to five years, you probably haven't heard of me. But I've published a book and I've done a bunch of other stuff and I've done a ton of open source contributions. So I'm kind of a fairly hardline FP person at this point. And, you know, we do FP for real. So we have a whole bunch of systems that are in production. So, yes, we use free monads. Yes, we use co-products. They actually do work
And so, yeah, that would probably be how I would position myself. Hi. Hi. I'm Roy. I work for Netflix, a small online subscription streaming company down south. I manage the Inside Engineering Group, which is responsible for real-time operational insight for Netflix. I'm here because, weirdly enough, I would say that my team probably does not identify itself as a Scala team except for the fact that they use Scala. And it's something that happens sort of naturally and it's something that we are, I think, relatively sort of non-dogmatic about
So I think the conversation I had with you, in fact, was that I might be the one person on the team who's not a hardline Scala person. So the tomatoes should be going my way probably. Thanks, guys. So I'll ask you a couple of leading questions, but feel free to basically have back and forth between yourselves. So one question I want to start with, I think Scala is extremely diverse. And so at some point, Hockey RT coined this term that Scala tribes. And kind of that, and I think we had a panel on the future of Scala with Rod Johnson. Right? After he basically at Scala Days, New York, he said, guys, simplify this stuff if you want really to have an option
Right? And so we had a lot of backlash. So we invited Rod and had this nice panel a couple of years ago. And Paul was there. So it's kind of, it's been, and IJ was there. So it was, I think we'll need to, because I think we kind of plan to meet in 2018. So we'll have reiteration. But the tribes term means, you know, we have a Java tribe, we have a Haskell tribe. So I wonder what do you guys think of Scala tribes? And if this model can let you kind of select organizational tribe, if this can help in any way to scale your teams
So I'll give that one a go. So I guess the way I would position it is, I guess I kind of don't really agree that there are completely discrete tribes. I would say it's more like a journey, in the sense that, you mentioned that you were coming from Java. So, you know, likewise, I'm sure in yourself, in the last few years, you've changed the way that you program in Scala. And that's like really, really common. Like if I look back 10 years, it's definitely what I'm doing now. It's not what I was doing then. And so I would kind of position more like, rather than they're like silo tribes and they kind of don't mix
Like, you know, be like, yeah, we're kind of doing Java without the semicolons versus like, you know, yeah, we only do three monads. Like, you know, these are obviously two opposite ends of the spectrum. And I would position, and indeed what we see with our team. So four years ago, we were, or four and a half years ago, well, four and a half years ago, we were 25 people. Now we're like 500. And so, you know, when we see this like acutely in the team, like, you know, people learn over time and they get better and we make harder abstractions more accessible to them. And we have libraries where people may use, say for example, free, and they don't even know they're using it. It's like, you don't have to know this stuff to use it
And I think that that's the difference is that's why I would kind of position it more like, you know, we have all the colors of the rainbow rather than, you know, all the separate tribes. I kind of don't like to think that people are like hard line one way or the other because, you know, at the end of the day, it's a technical subject. And, you know, therefore we can be objective and it shouldn't be about, you know, popular voting or anything like that. It's a technical objective fact and we can discuss, you know, technical objective things. And so, yeah, I would probably generally refute the tribes. I think I'll add to that quickly as well. In terms of maybe this is not a very popular thing to say, but I feel like most programming languages, the way I look at them is more like they're tools in your toolbox and each one is appropriate for something. So maybe you wouldn't pick Scala as a brand new startup if you want to be very agile and want to deliver something very quickly or a prototype that you might be doing that might not be the right use case for it
But it might, it'll be right for some other things. So instead of thinking of it as a tribe, I would totally agree with you there. It's the right tool for the right work that you have and it's more of a, more of a journey. And I would look at it more laterally than taking slices out of each one and saying this is a Java tribe and a Scala tribe or a Python tribe. Yeah. And I would say also that that's probably pretty relevant to the way we think about things. Since we're talking about scaling teams, Scala teams or otherwise, I would say that it's worth noting that we are generally actually relatively cautious when we meet a candidate who's particularly attached to their tribal identity. In fact, I was just talking to somebody today who has a fantastic candidate, but that candidate really, really, really loves Erlang
And there's any Erlang fans here? Okay, nothing wrong with Erlang. But my point is this. I think that maybe you could look at a MetaTribe, just to randomly make up a word, that basically separates people who are, in fact, attached to their language of choice versus people who are much more utilitarian. When the engineers under me chose Scala, they didn't decide that it was Scala for everything and they're not particularly attached to it. It's just the way they're solving problems right now. And neither of these approaches is wrong, right? I mean, I think that if you really, really absolutely are committed to working in Scala, find a job that will give you that. If you're interested in solving problems irrespective of the language, those jobs exist too. Yeah, I would just add that you kind of started business goals
You want to probably translate that with business values, cultural values, translate that into engineering cultural values. And then you look at that, you know, maybe experimentation, moving fast, or maybe resilience, or maybe being able to prove something is really important. And what I like about the Scala ecosystem is that for any one of those engineering cultural values, there's a number of really great open source libraries that you can pick and you can choose from. And so I really like the diversity in the Scala ecosystem. But just to echo what you said, you know, when it comes to scaling the teams or looking at if some library or some approach is going to work, I like to test drive it against cultural value for the team and say like, does this gel, does this job, what we're trying to do, and who's going to champion this? And is somebody else from the external like really telling us to do this, or they really are passionate about just this one thing, or does that really help align with our approach to solving problems? It can work either way. Yeah, I just want to add that, well, kind of all my colleagues on the panel here give a very kind of nice answer, but I just want to call it explicitly that there is definitely the wrong tool for the job. And I'm more come from, you know, this space that now referred to as data science, even though when I started it was not referred as data science. And for any one of you that tried to basically do data processing in imperative languages, then, you know, we can argue about what is the right functional paradigm, or sorry, what is the right functional language to solve kind of data problem, but it's kind of if you ever use Java to write MapReduce or God forbid to actually write Spark jobs, then that's probably not the way to go
Thank you. So I want to kind of maybe ask for your, to share some of the wisdom you guys have hiring people, right? Because, right, we, like I think it's a very unique situation, you know, all of the sponsors of Scala by the Bay are hiring. Most people who come to the Mid-Alps are hiring. So, and I hear a lot of different opinions on the, a lot of small staff are recently asking me, you know, should we just all move to Lisbon or, you know, Malaga, right? Because we are priced out of the market by Salesforce guys and Apple guys and Netflix guys, right? And, and, and, but for the large companies, it's also hard because, right? Because they need a lot of people on the teams and, and it's kind of, everybody is in the same boat and on, but you, you see folks who hire successfully, right? You see, again, there are small startups funded very recently and not, not very much, but they are able to hire brilliant people. So, if you can share your secrets, how do you convince, you know, the first engineer to join your startup? How do you convince a senior tech leader to join your larger company? How, can you share, how would you kind of persuade folks to join your teams? How, can you share your teams? At an abstract level, I think you've got to figure out what makes you different and attractive to your audience and you go with that. And ideally, that's something that is actually unique to you rather than everybody else. So, you know, Salesforce has some things going for it. And it would be foolish for Netflix to compete with Salesforce on its own terms
Twitter has some things going for it and it would be foolish for us to compete with Twitter. We have some things going for us and I'll talk about that, but generally speaking, you probably don't want to compete with us on the things that make us, you know, strong. Because chances are, we're, you know, we have more of those resources. So, for Netflix, what ends up being attractive to people is the tremendous amount of autonomy. We have a culture that is explicitly focused on context, not control. We talk about freedom and responsibility. And I can talk, when I talk to engineers, I talk to them about actually owning a product. Now, for some engineers, that's bad news
You know, if you're not interested in actually talking to your customers and figuring out sort of like real product management for your internal products, then this is not the job for you. But for the right people for our organization, that's actually remarkably attractive. The idea that you'll figure out where your product needs to be going. The idea that you'll figure out the right balance between tech debt and features. The idea that if you want to basically, you know, spend the next quarter focusing on tech debt, because that's the right thing for the long term. So far, it seems like that's pretty attractive to, you know, a reasonable amount of really, really smart engineers out there. So, as someone who's, you know, lost candidates to Netflix or gone for people, you know, they were interviewing at Twitter at the same time as well. And somebody who's, you know, currently looking to staff for really early stage engineering team
At those times of growth, I think you're trying to match somebody's aspiration. It's a custom fit. One at a time. It is not a repeatable process. It is not a scalable process in the beginning. It is really like one off custom fit that you're trying to find. And the more you can tap into that aspiration, the better off you're going to be. Rather than giving my own example, I'm going to give an example that I really think is great
Several years ago, we organized the meetup. There was a young person at the meetup. Driver was hiring. And they decided to sponsor Stuart Stuart, who's a great guy, to actually go and learn a lot about Scala, functional programming, reactive stack. And they really invested into him. They put him in a boot camp and gave him really great projects, put him into meetups. And so they took somebody who was really, really young, who was really passionate about learning. They're passionate about solving a really big, socially good problem to solve
And so they matched those two aspirations in investing into somebody for long term. And today when I talk to Stuart Stuart, this is a pitch for driver. I'm not from there. But they're a really, really great company when it comes to the mission and to finding somebody that was really passionate about learning. So likewise, in my own startup, long term stock exchange, if you guys are looking, you want to talk about the custom fit and solving a really big problem for a really great mission, disrupting the capital markets, I'm hiring and I'm available to give you that fit. That was a shameless plug. That's right. You can do it
I agree. I guess, so I completely agree with you about the custom fit. I guess I would kind of just augment that with, it's about understanding what people ascribe value from. So, you know, specifically, some people, they derive value from learning new things, internal value. Some people ascribe value to, you know, helping others, like an external value. And so you need to understand that, right? It's like you have no hope of hiring anyone if you don't understand what they care about. And, you know, likewise, you have no chance of retaining them if you don't understand what they care about. And so I would probably also add that I think the biggest thing for us in addition to what's already been said, which I think is true, is you need to have an environment where it's okay to fail
Like the bottom line is that if you don't fail, I mean, I heard something recently which I really liked, which is there are only wins and there are lessons. And, you know, if you don't have an environment where it's okay to have lessons, then you're basically fundamentally flawed. And we make so many mistakes overall. If you look past the last few years, we made a lot of mistakes and we learned a ton. So, you know, and from that perspective, I'm thankful for our mistakes and you need to have an environment where it's okay to do that. So, I'll say two things. The first one is also similar to, you know, there are no real tribes or, you know, this dichotomy between, you know, the function programming people, the, you know, Java without semicolon. There is also not so much as a dichotomy between companies is, you know, people kind of say, you know, small versus big
But for example, when I think, you know, I'm definitely worth for a large company, but honestly, and, you know, you can test it with people from my team who are here. Like none of us is really using cell first day to day. And I imagine, Tim, none of your people really ever climb in Verizon antenna to fix it, right? And so it's really about kind of what Roy said is really understanding what is your value proposition. And it's never actually like, it can be aligned with the company, obviously, as, you know, as a smaller developer position of the team and the company are aligned as you get bigger than you kind of have to find and kind of define your own. But it's basically, especially as you come to kind of a higher level and this is my now full-time job is basically growing the team is always be selling. And also there is like even for our before at LinkedIn and friends with Google and Facebook and all that, it's never easy because you always are facing if you have someone that usually is good enough for your team, you're probably good enough for five other teams. So you always kind of, you know, start from the Google with everything else equals like 20%, 25% of landing that candidate. And this is kind of where you need to go from
So just one last thing is what actually got me into data science is I read the book Moneyball, same goes for hiring engineers. So I want to add to that quickly. So definitely my beliefs resonate with everything that's been said here. And just I think on a higher level definitely when we think about hiring, it's probably the most important and the most expensive exercise any company does. Because as an engineering lead or as a manager, you're constantly thinking about how to grow your team. And I think from personal experience, I feel like a lot of the companies do really well at the start where the value proposition will be there. You'll try to find a good fit in the engineers that you're looking for. But a lot of them drop the ball at when someone actually joins and the integration part of it, the training part of it, I feel like that I almost consider a part of the hiring
So from the point you interview someone to the point they're actually integrated in your team three months into it or six months into it, I consider that definitely an important period, which not only makes a difference in the teams that you're cultivating, but it's word of the mouth too. They tell other people how easy it was for them to get started where they were. So, yeah. I'd maybe actually go even further than that. I think that's exactly true, right? Yeah. We talk about hiring and I've definitely had people, for example, like congratulating me when somebody like accepted an offer. And I was like, well, it's not done yet. And it's not done until their, you know, first day at work
And it's not done until they're onboarding. In fact, I would actually argue that the only time you're done with somebody is on their last date. Until then, basically, it's like an ongoing job. I mean, it's like, it's like giving birth. Like giving birth is not the end of, you know, having a kid. It's just the beginning of your next 18 years. Or in my parents' case, 25. I love this metaphor
Maybe I'll invite you guys if you have a question for the panel, please. Line up next to this mic. I'll ask one more question. And I'll call it, you know, there is a movie, right? Seven Private Ryan. I'll call this question Seven the Brilliant Jerk. And it will piggyback on the talk which Diane, so Diane Marsh gave a talk at Scala Days from Netflix. And she made this point, you know, fire the brilliant jerk. It will be good for your team
So basically, that was her, you know, crystallization of her experience, right? And I kind of felt a pang of pain because, you know, I could easily be that brilliant jerk in, you know, years before. And then I've been kind of on the other side and I kind of see, you know, being a senior guy at the company, I can see what it means, right? But I kind of, I still feel that it's a waste, right? There is the brilliant person who may, and I think it's really related to the functional programming context because a lot of people kind of might be mathematically inclined, right? They can fall into this. Again, I'm generalizing. But I wonder, from your experience, what can you suggest that can save the brilliant jerk? Can something be done? Yeah, okay. I have a good one for this. Or at least I think it's a good one. So in the case of, so I can't talk to whether or not someone's a jerk or not. As you say, I'm probably also that guy sometimes
You know, I think it's more along the lines of, it's about being uncomfortable so that you learn something. In the sense that, so if you're talking about how to manage that person, then I would probably position more like, okay, so you have someone who, I don't know, maybe they've been doing Haskell for forever or something. And, you know, they know an awful lot about FP. Well, great. That doesn't mean they know about distributed systems. That doesn't mean they know about, you know, low latency database access. There could be so many other things that they don't know about. And so I think it's important to understand that everyone has strengths
And putting people in a position where they can learn about their weaknesses and get better is a great way to manage people who are otherwise kind of getting above their station, so to speak. Because, you know, like the biggest thing I've kind of, just myself, I love it when I don't know something. And there's a ton of stuff I don't know. And, you know, like this just the past week I've been doing C++ for the first time in like, I don't know, maybe seven or eight years. And so a lot's changed. And for me that's like humbling because it's like, yeah, great. I don't know about that. And I think it's really, that's a good way to manage people is like to put them in a situation where they can learn and be humbled by the learning once again
Well, to be perfectly clear about my biases, I think you should burn brilliant jerks with fire. And I completely agree with Diane and it's not just because we're coworkers. So, but the question you asked is, you know, how can you save a brilliant jerk? This goes back to performance management, which is, you know, you ask any manager, it's like one of our favorite things to do. Nothing. I generally believe that there's a series of questions you've got to ask when somebody is basically not performing well. And the first one is, do they acknowledge there's a problem? Then are they, do they acknowledge it? You know, part of the problem at least is theirs. Then are they interested in solving the problem? Then can they figure out how to solve the problem? And then can they execute? And these are ordered, right? These are serialized. And the first hit no you hit basically tells you whether or not this is going to work out
And I would say that, look, it's okay to be a brilliant jerk if you acknowledge that you are, if you acknowledge that it causes a problem, and if you acknowledge that it's something that you actually want to fix. And if you're a brilliant jerk and you say, listen, this is a problem of God, I want to fix it, please help me. I'll work with you. Because I've seen brilliant jerks in those positions actually become non-jerks. Brilliant non-jerks. But the problem is that in most of my experience with brilliant jerks, those are the basically the inadvertent brilliant jerks, like the, oh my God, I'm sorry I stepped on your toes. I apologize, I would like to do better. There's a whole bunch of brilliant jerks with whom I've worked, who acknowledge that they were brilliant, and just basically said that being right is just their sort of, you know, form of being a jerk
And for those people, I think there's no saving them. And in order to tend to the ecosystem, in order to tend to an overall productive and healthy environment, you know, we talked about scaling teams. One of the things that you need to do is actually have your engineers be interested in coming to work in the morning. Brilliant jerks burn that. And I think you have to be pretty aggressively attentive to that. Yeah, I will definitely agree with that statement. So, one of my realization when I kind of became a manager and then manager of managers, is that honestly, every time I get promoted, my hourly wage goes down. Like I get a nice bump in salary, but then it's like the work double the time
And it's totally not worth it and overrated for those of you who ask. And one of the things I realized is just actually, like, there is just not enough time in a day to deal with brilliant jerks. And I recently had to deal with a situation where a manager under me had a person, I said, look, here's the analogy. We interview so many people, right? And honestly, most of the people we end up passing on for various reasons. Some of them, you know, we make tall mistakes, but no one ever thinks about it after, okay, you know, it's kind of maybe, I'm pleasant, you send the letter, thank you for coming, and then you just move on with your life. And I think we actually need to be more in that mindset, even with existing employees. Usually, I've never had, like, this point in my life because maybe I was just not privy to the details saying, oh my God, we fired that person too soon, right? It's always just too late. And if you think about any other relationship, and not just, you know, with a employer, probably it can only be too late
So I'm saying, like, just, it's probably not worth it. So I've been called a brilliant jerk a bunch of times. And when I look back at, you know, what I did in, you know, when people did that type of performance management with me, you know, what I realized that I lacked, and I would add to, like, helping humble somebody or helping them give an opportunity to learn something new and extend and reach out. I think a lot of us, you know, probably in this room have, you know, gone to a lot of academic training, either formally at universities or afterwards, or we read books or we do stuff offline. So we're, like, very technically astute, but very few people actually get any training about how to deal with brilliant jerks. As well as brilliant jerks get very little or no training ever, they're actually sooner fired than people giving them training, which I think is wrong. And it's really actually hard to sit down with a human being and tell them that they're, you know, fucked up. And, like, because it's hard, and human conflict is hard, the sooner you do that in life as a manager, the sooner you give that opportunity to somebody to sit them down as a human being and tell them that that's what's wrong, and give them some training, give them some opportunity, the better it is
Usually the way these things go is, like, you burn the relationship, you burn the bridge, and trust is kind of binary. You either have it or you don't. Once you burn it, you don't rebuild it. So you might have to let the person out, let them walk out. But please do reach out. Like, if you have a colleague who's been a brilliant jerk to you, against you, towards you, please sit them down and let them know. Like, that will pay forward so, so much. People don't get the training
I completely agree there. While I was managing, I think that was my first thing to do. If you come across, and I feel like it's very hard as a colleague to do this, but as a manager, I think it's your utmost responsibility to at least sit them down and tell them what is going wrong before you, say, get rid of them. It's more charged when you're a manager. More? charged. Like, the higher things are at stake. If you are a colleague, you might be less charged. Definitely
It's hard. You might not see it as your moral obligation or a professional obligation, but usually it goes way easier when you tell your colleague, your peer, and when they're being sat down by their manager. Yeah, definitely. I agree. Or maybe you can go the manager route and tell your manager to talk to them. Whatever is easier, but I feel like a chance everyone deserves. So it would be the right thing to do to tell someone honestly what's going on. And if, like Roy was saying, if they're not willing to change, there's not a lot you can do about it, but they might be
Maybe they don't see it. Yeah. just to clarify, I do not mean you need to fire a person, you know, if they didn't say hi in the morning to you, right? So, like, yes, assuming that, you know, everything else failed, I'm saying there is just a point that I guess I'm on the side that you should, you know, just move on quicker than the others. Definitely. Yeah. Cut your losses quicker. Thanks, guys. I think we have a question from the audience
So if you guys have more questions, please line up here and we'll... I wanted to ask about technical debt. How do you, in your experience working with Scala teams, do you handle and eventually break through that technical debt, assuming you prioritize it? And along those same lines, how do you prevent that from happening in the first place? Firstly, you need to accept that it is going to happen. Like, I feel like for a lot of people, they're like, we just made this brand new architecture. It's perfect. And, you know, we've all been there, right? The rewrite, the third rewrite, and then you're like, this time. is going to be perfect. And I think it's really important to understand that it's going to happen
It's inevitable. And so if you embrace that as a team, then you need to understand that libraries in Scala are the key battleground for technical debt. Like, libraries are like the complete and absolute devil. Like, so I know that sounds really bizarre, because everyone's like, yes, don't repeat yourself. Make this a library. And then, so I make it a library, and then, you know, hey, one other person uses it, and another person uses it. And then maybe six months now, I'm like, wow, there's this, like, gnarly performance problem. Let me just rewrite my write-by, because this time it's going to be perfect
And then I'm like, then I have to go through and update, like, the, I don't know, 200 services that are now using this thing. And so, on this note, actually, we actually open sourced a SPT plugin we wrote to kind of, like, try and mitigate this. Because it just, it's such a nightmare on the JVM. Like, you actually find yourself having to restrict certain dependencies so that it can't be used, like, either directly or transitively. And so, like, if you, once you get big enough, and in a, I would position more, like, depending on what it is, you want to kind of minimize your core libraries so that you have a core set of libraries that are maintained by one team. So there's a constant level of service. And, you know, I'm fixing bugs and the way that the APIs are and things like that. And then, in addition, to be honest, if someone's got a useful piece of code, don't make a common util
That's like a dumping ground. Like, anything that's called util just becomes the dumping ground. Like, oh, I got this util. Let me just put it over here in the util library. And then before you know it, it's like 20 megabytes. And it's, like, just got a ton of stuff in it that no one uses. And they're not really too sure it's there because it's been there for five years. And so, you know, we've all been there
And, you know, I mean, hell. But, you know, I would sort of say, so a lot of the time it's actually cheaper from a technical debt perspective to cut and paste code. Certain pieces of code, I'm not saying do a lot of cut and paste. Don't do that. But it's a, you know, it's a case of knowing when is it the right time. Like, if you're making a library, that's a key battleground for technical debt. So make a decision about when and where you're going to do it. Refactoring services is kind of like, meh
Like, provided you keep your external protocol the same, then you can rewrite it, you know, to your heart's content. But stuff that gets everywhere is a complete nightmare for technical debt. I want to actually go stronger than that. I would argue the question of how you prevent technical debt is really begging the question. Because it suggests that you should even try to prevent technical debt. Listen, I don't live in San Francisco, so I can afford to own a house. And because of that, I have a mortgage, right? So I have a ton of debt. And that's okay, because I'm getting something out of that
Technical debt is a useful way to get something done. So I wouldn't argue that you need to try to avoid technical debt. I think you need to properly manage technical debt as you manage any other kind of debt. For us, that's been a challenge for us for kind of similar reasons, because we have libraries. So my team builds libraries that literally every Java service at Netflix uses. And we had a problem where we suddenly noticed that basically when we release a new version of the library, within about a month, 95% of the services have picked up the new version of the library. And within two months, we have like 98%. And you can see where this is going
And then like two years later, you still have like three services using the old version. So for us, what we did was we actually sat down with all of our customers and we came up with a, frankly, an agreement or a contract that basically said every quarter will tell you a really well-supported version of the libraries. And you'll have a quarter to upgrade to those versions. And at the end of that quarter, we're okay actually breaking you. And here's what you get out of this process. Here's what we get out of this process. We call this the quarterly deprecation cycle. It's worked reasonably well for us in maintaining a certain level of ongoing technical debt
Yeah, there are levels of technical debt. I feel like I don't know how far back I want to go. But in terms of Twitter, our technical debt has been well-known. We've talked about it in terms of us having the monorail and everything was one giant monolith. And any time you made a change in Twitter, you had to redeploy all of it. One way we broke out of it was all of engineering. It became a goal for all of engineering to break out of that loop. So instead of thinking of technical debt as a backlog of tickets that you kind of chip away at slowly, it became a priority for all of engineers to tackle it head on because it was that big a problem
But not everyone gets to that level of technical debt and ignores it for a long time, or I hope not. So if the levels of technical debt, if it is not as big a case, I think the easiest way that we've looked at now, I think microservices definitely is a way to go just because you keep separation of concerns there definitely. And also in terms of just managing technical debt, as Roy was saying and other panelists here, that don't look at it as a thing that you shouldn't have. More look at it as it's going to be there. So find out ways that you would tackle it, whether it's a backlog of tickets that you chip away at every quarter and you have a key result at the end of the quarter that says, I need to get halfway through this backlog or however you'd like to manage it. But don't let it become as big a problem that all you're doing then is technical debt because even that's not a very healthy place to be at. You want to be doing your future work alongside of handling the debt. So instead of answering, maybe, Tijo, if you don't mind, I'd like to propose
There's many questions, maybe like two or three answers. We have more questions. Hi, I'm Luther Berzel. I founded a startup and prototyped and launched V1 in Scala. So a very different experience and perspective on the hiring piece in the adult world. Do you find that when your hires don't work out that it's because of cultural or technical reasons? And how do Scala-centric engineers compare to the broader population in that? I can try. I think when, I guess similar to technical debt, I think the assumption that 100% of the hires should work out is the wrong. It's the wrong assumption
Honestly, again, hiring is not Catholic marriage, right? It's just, some of them will not work out. And by the way, similar to, you know, like I had experiences personally that, you know, I just, there was not a great match between me and the company I've worked for. So, I would say, and I'll let someone else touch on the Scala point, is set what is the right assumption of what percentage is realistic. Because, you know, no one hits, you know, they're more amazeable 100% of the time. So, what is the maximum that you should aspire to? And I guess that is something that the number could change as the company grow and you can become better at, like we said, the onboarding, retaining. Like, for example, for us, the people is our number one value. And it says hiring, motivating, and retaining. So, actually, motivating and retaining is, you know, two out of the three things that we need to do
And as we become better at this, that number should go up. But it's never 100%. Right. But when, in whatever percentage it is that doesn't work out, is it because of cultural reasons or technical ones? I think that's the same. Yeah. I would say it's usually, it's usually performance. And I would say that's, for me, the main reason when people are not working out. But the performance, I would say, it's very rarely because the person is stupid
It's usually because they just don't have, I don't know, the right level of energy, the right drive. Maybe they're just not very excited about the project and that's okay. So, I would say that definitely the, it's more motivational than, I would say, that lack of skills, of knowledge to be able to execute on the job. I would just say the same cultural and technical, at least in my experience, in the sense that, you know, I had a couple of opportunities where I made the wrong hire. The person who wanted to do X, but that was not necessarily what we needed to do at the moment. And it, like, really quickly becomes about, like, well, I want to use this library and I want to use this approach. Or we're arguing in pull requests around, like, you know, adding extra, you know, functional this, that, the other. And when we need to ship something quickly, we do it differently
And so, it was my mistake, it was, like, cultural. It was, but it manifested itself technically where everybody was saying, like, why are we adding all these libraries? But it was purely cultural. So, I, like, how does that compare to other ecosystems? I think, you know, Scala ecosystem is big enough and there are many engineers in it that I would be surprised that it's any different in any other ecosystem. Thank you. Hi. I have a question about, so, we were talking earlier about Scala tribes and we said maybe it's actually more of a spectrum of FP. So, as an organization scales up, it would be natural for teams to fragment and you have some teams doing one thing and some teams doing things a different way. A lot of companies, as they grow, will have an architectural group that tries to impose standards to allow teams to read each other's code and interact, interoperate more, but that can stifle innovation
So, as a company grows and you have more teams, how do you strike a balance between having common architectural principles that everyone follows without stifling innovation? Sorry, please. Sorry, I was just going to tack on to the standards. Definitely, I think standards are good to implement, especially with the kind of community that we have. It's in the stage where it's still growing. It's, I don't know if I'd call it mainstream yet, but in terms of standards, companies like, shameless plug here, like Twitter, we've dedicated a lot of resources and time into thinking about what the standards should be and how published them. And I hear from a lot of friends who work in Scala that they do adopt those standards and it makes sense because you don't want to reinvent the wheel there. If there are people who have done the work for you, definitely take those and override them with exceptions that are specific to your company. But if there are folks who have done this work before you end up thinking for it, I would say adopt those standards
And tooling around things, I think as long as we can standardize that, that would make a whole lot of difference to just the growth that you talked about in terms of how do you, when something is growing, how do you maintain it so that it doesn't grow out of bounds or basically you have some kind of a check on things. When we talk about, Twitter has an architecture group as well. So there is a lot of thinking that goes into, all right, we have this thing that's growing, but we want to make sure that it adheres to certain standards. And we would like to open source tooling and documentation and books around this effort so that the rest of the community has something to hold on to. I guess I would, I agree standing on the shoulders of the people who went before you is a good way to go. I disagree with the term standards though. So the wonderful thing about standards is that everyone's got one. So I prefer it as a recommendation
So because we kind of form that, you know, the sort of analog of the architectural kind of group. That's why I sort of don't term them as a standard because the problem is that the standards are always changing. A little bit like what was mentioned earlier with this whole quarterly deprecation cycle. There's always something new. And likewise, the parallel I would give or, you know, something to think about would be, like in my team, for example, the things that we produce primarily value correctness. So the way that we program, you know, reflects that. But for example, there are other teams, particularly the teams that work on the services at the very edge of the world. Latency is king
And so those things, correctness and latency, on the JVM, it's like, you know, cap theorem. You know, I just can't have them together, you know, so I choose to. And often, you know, you can't get correctness and latency. So you have to make some trade-offs. And I think that it's fine to have recommendations about the way to do things. And I would say it's also really important to support people in, like, sort of a multi-pronged fashion. For example, myself, I'm not a classroom learner. I can't sit there and listen to a classroom
I, like, literally can't do it. And so for me, better, like, self-learning is better for me. Likewise, for other people, pair programming is an immense source of learning. And so, you know, I think recognizing that as an organization you need to have multiple support structures in order to educate and, like, spread things in a team. Like, it becomes less a sort of, like, doctrine as, like, here's the standard. Meet the standard. And, hey, here's our learning path. Like, in fact, actually, our recommendations today are for, like, X, Y, and Z
And, you know, oh, you don't know what that structure is? Let's help you. Let's advise you about how best to do that. Here are some resources you can look at. And I think, you know, because the ultimate thing I think we learned as we scaled the team up was there just isn't one answer. And that's been the really difficult thing to kind of swallow because it's like whatever you do is kind of wrong for someone. And, you know, I don't know if other people have had that experience too. But, yeah, we've kind of tried to tackle it simply by having multiple avenues of, you know, learning and support. And that's kind of an overhead
But I think it's a worthwhile one. I just wanted to answer because I didn't mean to say standard as in everyone should adopt the Twitter standard. I meant more along, like, code formatting tools and things like that, that the work that has been already done, of course, you can override it with your own special requirements. But also, I completely agree with what you said in terms of it's different for everyone. But the more open you are in providing that kind of training, whatever levels that's at, whether that's, sure, not everyone can have a scholar school like Twitter does. But even at the level of whatever your size company is, I worked in a startup myself for a few years. So I totally know the resources you have at hand when you have, like, a five-person team. So the training, as long as it's in the mindset of people that you help them out, then it doesn't matter what level of the team that you're in
So I'll just add my, quickly, two cents here because I moved from a company at LinkedIn. You can use any language you like as long as it's Java. And at Salesforce, if you, you know, watch the news, we're fairly, we have an appetite for acquisitions. And so we acquire a lot of companies. And, you know, some of these companies happen to be in the same stack that, you know, at least I'm saying that in the, my team is using. Most of them are not. And then we have this question, okay, you know, should we try to rewrite that technology to use our stack to get leverage and all the wonderful things that come from, again, standards, recommendation, tooling, and whatsoever. And the answer, kind of, the answer is always the same for us is what is the business value? What is the value to the customer? And so that is when we take a company that comes from a different technology and talking about rewriting it
But the same should be, you know, when there is a team that says, hey, I want to actually go off and build something new on a new stack. That's great if there is a value, a real value to a customer. And when I say value, the value should never be just our engineers can develop it quicker because it always has to be with, you know, hey, what happens if this product, God forbid, succeeds? And actually you need to support and maintain and do all of this thing. So, you know, if the answer is still yes, that is, you know, a new technology is the right way to go, then yeah, that's probably the way you should think about it. I would just say that as teams scale and as they work on different things, communication and standardization of any sort, the recommendation becomes harder and harder exponentially. And what you learn to do just by trial and error is that every message you do, everything you want to adopt, it has to be like redundant and available in many ways, how you communicate it, how people can get it, how people can integrate it. You know, whether that's, you know, providing scholar school or meetups or referenceable architectures or anything like that. But you need to like, if you want something to take hold, as you grow the organizations, you just have to do it multiple times in multiple ways
It's a lot of work. I think we have time for one more question. Hi. So I actually had a similar question when I was standing in line, but he asked it. So I want to have maybe a sort of the next level on it. So assuming you agree that you're going to have some standards or consistency that you want to enforce, how do you recommend doing the enforcement or, you know, checking whether it's being adhered to? So is it strictly automated tools? Is it code reviews? Is it combination? So that's sort of at the, you know, just general consistency with libraries used and idioms and stuff like that. And then the second question is specifically on code formatting. Do you enforce consistency via tools or how do you do it? So, please, I'll give it away
I think it was kind of an issue. Okay. Thank you. So dependencies, as I mentioned, like libraries are super hard. So we have this thing called SPT blockade. It's on GitHub. You can check it out. It basically allows you to specify as a central authority
You can be like, this library, let's just say it's like something evil, like, I don't know, juice, that you don't want it to be used. These are all banned, by the way, in our organization. Any dependency injection framework is banned. So basically, if developers try to use them, they get a huge warning in their console which says, you're using a banned thing. And then when it goes on to Travis to build, it fails. And so this is pretty awesome, by the way. And so the reason I bring this up is because for internal libraries, so for example, like, I don't know, let's say we produce a new version of like some internal library. We then basically set a deprecation cycle
And then because the blockade is applied to all builds, they get a message. Let's say we give them a month to deprecate an upgrade. And then they'll get a message every single time in their console to be like, yeah. Or, and it's actually really nice, we just added this. You can actually say like, oh, you're depending on the library which depends on something that is banned. And then it will sort of do all this interesting stuff. So absolutely, do we automate things? Hell yeah. Because like once you get to a large scale, or even a smaller scale to be honest, like, I don't know about anyone else, I just tedium of like dependency management is just like unbelievable
So, but code formatting and stuff, to be honest, I don't really care about that. That's more of a religious thing I feel like. And, you know, people like, for example, me and Daniel Spiwak, we like constantly argue like, he hates the way my code looks and I don't really like his either. And like, you know, it's just like one of these things, you know, like it's just a formatting thing. Like if the code executes fine, it's like provided it has a particular, you know, style. And so typically, because we have different groups, the different groups kind of, they have their own thing that they agree on. And if that enables them to get their work done, more power to them. Yeah, it's more of a problem when people within the same group have different opinions, and they're working on the same files, right? So
I would say that actually, I come from an anti enforcement culture. So, you know, going back to the original question that I didn't opine on, we tend to basically say, here's the paved road, you can choose that or you can choose anything else. And we deal with duplication and inefficiency, sort of in a lazy binding level at the end. And we tend to bias really, really, really hard against enforcement, frankly. There are some things we'll enforce. The people working on our, you know, credit card processing systems are operating in very different environments than the rest of us. But otherwise, we find that that enforcement basically slows everything down in a way that, frankly, we're not really willing to pay for. And makes innovation much, much harder
Because essentially, you're going down the road of saying, the only innovation that's going to happen is sort of like the company sponsored and the company approved innovation, rather than people doing the random stuff that you haven't even planned on yet. I'm not sure I totally agree there. But it's very interesting to hear a very different point of view. I mean, at Twitter, we have standards for code, from code formatting to code reviewing, everything has a standard. And I don't think, I've changed three different teams at Twitter, and I feel like I've been fairly efficient every time I've made the switch, just because everything was very familiar. I didn't have to relearn things, because everyone had their own way of doing things. On top of that, when you are at that size, I don't see how you can't automate. That is a key to enforcing anything
We have long deprecation cycles, but even to the point when you said, warning messages get printed, people ignore them. Yeah, they do. That's why the bill is. Yeah. So, then automated tooling and enforcement by actually failing your bills and actually blocking any checking at Twitter, you cannot, master is always green, so you cannot check in anything that's breaking the bill. Sure. So, if you remove that barrier, I feel like it would be chaos. So, maybe it's a size thing as well, because of the scale that we are at, but definitely we've got a standard around code reviews, code formatting, builds, RCI
And I think it's fair, right? We're at a different scale. We only have about 1500 engineers. But, I would say that... And 2000? I don't know. So, to be perfectly honest, I'd love to argue for like 30, 45 minutes on this whole topic, and we probably shouldn't right now. Thank you. I think we have just one last question, which is especially waiting, so we'll ask that. Hopefully it will be fun and fast
Perfect. Thank you. So, coming back to scaling teams, what has been most successful or unsuccessful with regards to internal training? You have to bring up people from somewhere, right? What if you don't hire somebody with existing scale experience, they know, God forbid, Java or... I think that's a benefit, actually. So, I reflect on this for some time back in 2014, and there's this interesting thing that I realized that... So, I'll give you an analog, right? So, for example, like a... For example, if you have someone who's been doing Java their entire career, the probability is that they actually really understand the JIT and profiling and these things extremely well. I mean, it's possible that they don't, but in my experience, a lot of the people who've been doing it a long time do
And so, they bring a value there, like operationally, that they actually understand how things work at the low level. Likewise, if someone's come from a different language, then great. They don't have any preconceived conceptions about the way that things should work. And so, I think it's about taking the things that are... So, the biggest benefit, take the things that appear to be a weakness and make them a strength. And that's a really great way to succeed. But in terms of what's been difficult, I think that one thing I would definitely say, just to be quick, when you tell someone about a structure, like Clisley, for example, like, it's like being shown the path and like walking it yourself are like two totally different things. It's like knowing what a monad is doesn't mean that you can like, sort of see them in your own problem space
And that's something that, best we can tell, it just takes time. And the simple passage of time, they just get better at seeing those structures in their own problems. And, yeah, Ryan, who sat down here on the front row, did a nice talk, was it a year ago? Back at Clisley. Yeah. And how he applied Clisley to his own problem domain for DSLs. And that's something that once you get deep into something, it's like you can see it everywhere. And it's like a lot of these things that you just, you really need to use it a lot to see it. And so that's a challenge
I'll just give you one anecdote. I had a new engineer start on last Monday, who doesn't, who didn't know Scala. In fact, when we interviewed, he was like, so can I, could I rewrite my stuff, my stuff in Java if I want to? And I was like, sure, no problem. It was just an example. Well, but here's the thing. He joined last Monday and I sat down with him today and I said, how's your week, man? And he was like, it was great. I learned Scala, I wrote my first production service and I deployed it to production. I'm not saying he's the world's greatest Scala developer
It's quite possible that almost everybody in this room is a better Scala developer than him. But I have zero doubt that people can learn Scala. Honestly, I don't mean this as an insult, but it's not that hard. So we don't hire for Scala knowledge. We hire for greatness. We hire for the ability to learn. We hire for enthusiasm and we hire for, you know, frankly, faith that it's going to work out. And so far, frankly, we've not been disappointed
I think I'm going to just reiterate that. Another shameless plug for Twitter, but in terms of we have something called Twitter University. And I remember, I think it was definitely about five years ago when I joined Twitter, that I went to Scala school because I was a Java programmer. So I'd never worked on Scala before that. And that was the way I started working in Scala. So definitely, if you're coming from another paradigm, imperative, it doesn't matter. It is something that you can learn. And as long as you have the environment to learn, people who are good at a programming language are not even and have the ability to think and logically work out problems
I don't think it's that big a deal. All right. On that great note, we want to thank all the panelists for your insights. Thank you.