Scale By The Bay 2019: Thursday Keynote, Heather Miller, The Times Are A-Changin'

Scale By The Bay 2019: Thursday Keynote, Heather Miller, The Times Are A-Changin'

Recording: Scale By The Bay 2019: Thursday Keynote, Heather Miller, The Times Are A-Changin'

[Music] good morning so as Alexi said I'm a professor in the school of computer science at Carnegie Mellon University and I do research at the intersection of programming languages and distributed systems and I actually would love to talk to you about a number of research projects that I'm doing but it's also really when I'm addressing a whole group of people and everybody is very different we have different backgrounds you know it's talking about my research that maybe only twelve people in the room might be interested in is kind of like a waste of everybody's time so I'd like to talk about something else that's really I think super important to all of us and that we should maybe try to pay attention to especially in the coming couple of years so as Alexi said I'm an assistant professor at CMU I sort of co-founded the scale Center with Martin or Derek's key in 2016 I did my PhD with Martin on Scala I did a bunch of things related to Scala like you know I worked on futures and the concurrency libraries wrote a lot of documentation I worked on the Coursera MOOCs I did a lot of stuff related to Scala so I you know I'm probably responsible for something you're pissed off at and I'm sorry about that if I ever did do anything that annoyed you and I joined CMU as an assistant professor in 2000 in 18 the sort of things that I'm doing at CMU is trying to bring techniques from programming languages to distributed systems and particularly I'm working on about three projects right now one is about making Microsoft's architectures actually more reliable and convenient to use so like you're using one programming language even though the whole point is not to use one programming language and we work a little bit on distributed for actor runtimes and I'm happy to talk about all of these things but please talk to me about them offline if you're interested in any of these things you know I I would love to chat about them but this isn't the topic of this talk and I'm working at CMU with some great folks you might know Chris Michael Johnny's Esha on the Conny student called Matthew Whedon are working on CRT T's and if you're interested in the PhD come talk to me but sort of the the the gist of this talk is you know in 2016 I was or 2015 actually I was graduating my PhD and then I was like okay now I have to be the executive director of this thing that nobody even knows what it's going to be yet so what do we make what do we create and you know really the most important thing at the time was like okay you know there are problems in the scylla community not just with like you know community governance but also with like how do we keep you know libraries alive and how do we you know keep people who are working on these things how do we like fund the most important parts of the ecosystem that everybody needs so my focus at that time become you know I was like hacking around and on you know programming languages the compiler e things and then suddenly I had to shift 100 with 200% to sort of open source Scala ecosystem things like that how do we improve the developer experience for everybody and not just the people who are paying into the scylla Center but anybody who has an internet connection they deserve a good a good sort of developer experience especially for Scala they want to learn our programming language it should be not ridiculous and terrible for them to try and learn it and so when I had to suddenly shift from like you know whole bunch of technical stuff to sort of you know looking at community ecosystem you know I was going around talking to various companies a lot and I learned a lot of things that you know I couldn't really articulate and like share with other people because I didn't really have data points or anything to share with others but you know it was it was a lot of things it was not just like health of open open source ecosystems and like how to you know sustain stuff that seemed to be a problem but you know there was also problems with onboarding new people all of these things I mean it wasn't just Scala only right it was like everybody has these problems so I I noticed that this wasn't just a thing because people from other programming languages communities would come to talk to me about it because you know like I'm good friends with the people who run rust and they're you know we share tips all the time about like weird stuff that goes on and we're like you know you did this well like how do we learn from you and then they come to us when we did something well and they're like well how did you do that right so we found out that like a lot of these things you know that that seemed to be like my problem as the person in charge of the skeletons were actually like really most people's problems and so that brings me to the topic of this talk I started actually at that time in about 2015 collecting a whole bunch of data just like every time I would find something and you know that sort of dived with some of the things that I was observing and basically it's these three topics that I think are important for all of us is just like you know like make a mental note of and walk away with it the first thing is that in like the last five years how we actually build software has changed I know that a lot of people make these like you know sweeping statements I have data to back it up the you know the other thing of course being the Scala Center person I'm like oh yeah stuff about open source is changing as well there's things that we all depend on that we should be sort of aware of things going on in that world and also what we think a software engineer is like when we want to hire a new software developer actually like what most people in this room thinks a software developer is you're all wrong because it turns out that there is this massive influx of people that have almost no experience coming in at like a rate of ten times like the number of people that like you know are in this room when they came in right it's insane and so I'm gonna touch I basically want me to do is going to go through and provide data on these three areas and then I'm going to kind of conclude with some observations that I've been able to derive from these things and I think that they're important for all of us to walk away with and the first thing you know okay how people get into these jobs is changing we all know that hiring is not easy like if you work on software you know that it's hard and everybody writes articles about it all the time but actually I went and collected data from so this is a graph that I made based on data from the department of the educational statistics and so there's other data superimposed on this that comes from the Bureau of Labor Statistics and again this is so I should point out this is like you know us data so like if you came from abroad this may be it does not match with what you see but the point here is in so this is this is basically the number of computer science degrees that have been awarded every year and the the data actually this is the most recent data point because they have like a three-year delay on publishing these things and the point here is that if you look at all of the universities in the US that are graduating people with computer related degrees so like information technology degrees computer science degrees computer engineering degrees we're graduating around 660 thousand people per year in u.s. and currently the rate of growth is about 7.4 percent per year so the blue is like the number that the years that we actually have numbers for the red is basically based on this this projected growth rate of about 7.4 percent per year what it should look like and then okay well you know what about coding boot camps those things exist right I'll show you that in a second but the Bureau of Labor and Statistics published this number and I think 2017 they say that there are 500,000 open computing related jobs so this could be application developer this could be a system administrator this is basically things that are broadly under the category of computing jobs in the u.s. right now 500,000 and so like however I want to stack these these things you know it's it's kind of hard to like you know fill these jobs these are the ones that are currently available supposedly and if I add into into these numbers the people that are graduating from coding boot camps because these would be also entry-level developers that we could potentially hire this is what you know they've recently started publishing actually pretty good statistics about the number of people that are graduating from these things every year and the oops the the growth rate is 14% so I can stack them on top of you know of these of our CS degrees right and then we can see that you know in 2026 we'll have about just under 200,000 people that could have potentially in the u.s. be edging or you know get an entry-level computing job is the idea right but the reason why I go to 2026 is because the Bureau of Labor and Statistics projects that by 2026 the number of that number that 500,000 number is going to grow to 3.5 million and that actually takes into consideration the people who are graduating every year so this is kind of like an unfair graph because if you've graduated like you know this takes into account the people who have gotten degrees and then gotten jobs so we're still supposed to have some ridiculous deficit in a few years and they say oops they say that only about 19 percent of these jobs are supposed to be able to be filled by people in the u.s. who graduated from American degree or universities just you know assuming everything stays the same right now so I mean I you know we can like see em use a fancy private school that's super selective like we could just say hey everybody like let's let everybody in and I'm trading more computing people right that's still not gonna fix this problem and I mean it's not just like some like Bureau of Labor and Statistics numbers that I'm showing you and like you know every everything is all about those numbers you can actually see some of these trends in in in you know on the Stack Overflow surveys so this is not exactly a perfect comparison to the slide before because if you look at these numbers they're all like different buckets but basically what these things show is the number since 2017 2018 2019 Stack Overflow professional developer developer survey results and the question that this I'm showing you is the one where you know if you have a job how many years have you been coding professionally and basically if you look at this one 2017 you could say about 50% of people who answer that survey and we're talking forty thousand people have up to five years of experience fifty-seven point five percent of people have up to five years of experience in 2018 so that number increased and so this is hard to compare with the other ones because again the buckets are a little different but basically people you know 41 percent of people have less than five years of experience but the whole point is that if you look at this you know and these are you know on the order of tens of thousands of people responding you know a vast majority of all of the people responding to these things have only been developers for a couple of years everybody in this room at least most of the people I've met you know you're all practicing for more than five years a lot of people in this room are so the point here is that you know we've got a lot of newcomers and in particular we got people with only one or two years of experience and that number is expected to grow dramatically and like more dramatically I think then we can kind of comprehend and yeah we can publish more tech articles about how it's hard to hire but like I don't know what we're going to do about this problem so you know a bottom line here is that the demand at least if you want to believe all the PhDs from random fancy places like Harvard and economics talking about like projections and the economy if you want to believe those people it looks like you know the demand for developers is probably going to get even more ridiculous and more ridiculous and we can probably comprehend and if you look at you know even just the stackoverflow surveys the average years of experience of software engineers is going down overall every year so what we really have is this tidal wave of newcomers entering our profession and you know according to the government anyway this is supposed to only just pick up speed and you know we can't just do what we normally do like we can't onboard people the way we onboard people right now if we're you know if this is actually going to happen and we're not going to like you know fail at it so one thing that's really nice is there's a guy who works or who basically works on a coding bootcamp in Tennessee um he makes this point that I think what we can all kind of agree with which is basically I mean you know we've all had to write software in the last year I hope so this is not surprising to you but you know it's meaningful to people who are newcomers right though the point that he makes is that new frameworks are lowering the barrier to entry he says that's a far cry from the days when you had to learn the syntax of several programming languages to build useful software rather than typing these seven lines of code to get a menu to pop down you just download the framework from a code base that allows you to do that in a simpler way frameworks are taking the hard work that developers prided themselves on out of the equation so that's a good thing right like so all of these frameworks I mean that will make fun of all the time because every day there's a new framework they're actually making it easier for new people to just do something useful and meaningful more quickly than you know when we had to learn back some more years ago and another thing that's happening so I'm post I have these slides these fight slides are currently posted so all of these data points you can actually go and look at up here if you don't believe me there are a number of other there's like a movement of other articles basically claiming that existing developers are printing out I know a lot of people who have burned out and actually changed professions and I think that you know you all probably know one or two people as well so unable to fill tech vacancies employers shuffle off additional duties to current employees which leads to burnout and has a negative impact on local business development over thirty percent of respondents of a survey done by indeed comm admit that the issue accelerates staff turnover so basically people are quitting their jobs because people are there's their shortages and people are putting more tasks on the few people that they have those people are burning out and leaving with companies unable to fill positions current employers are expected to fill the gaps in many cases this results in turn her over over a third respondents of this indeed survey said the lack of timely hiring has caused burnout and existing employees and affected their businesses I mean again this is a thing where you know it's hard to have like data to show this but I think a lot of people we all know somebody who burnt out because they had a bunch of responsibilities on them right and this is like you know everywhere it's not just like your company is bad and you're burning people out like all of us have this problem so obviously I could make the point here that increased diversity would help right we know that the people who are making software aren't always a representational sort of sample of society so we could probably try to increase the number of people who make software by increasing diversity that would be good also immigration if we actually the u.s. got less bad at this we could probably fill some of these jobs and then there's the whole point about remote workers right like you know maybe we should be a little bit more open to this because it's gonna be really hard to get people to do these jobs but you know fine I can say that we have a diversity problem but like what do we actually do like we actually have to do something about this and there's nothing that you know like I mean what we're gonna wait for the government to do something know that the government doesn't do anything in a timely manner right so you know we can all individually actually do something and that is actually caring a little bit more about mentoring because if we had this onslaught of new people coming in all of you all are gonna have to get much better at mentoring other humans in order to not lose those new humans that come into your teams and do I know how to mentor people well I'm not gonna claim that I do I'm just going to say that we should all get better at it but yes again diversity immigration and you know basically embracing remote work these are also other things that can be done at a larger scale that you know us as individuals can't necessarily control but like we should maybe all focus a little bit more in mentorship and so I'm not going to talk about so I've got a few slides on diversity I'm not going to talk about it I'm just going to like leave the slides up here so that you can look at it if you want to later but this is actually a real research study that you know has quantified and you know the difference of diverse teams and found that basically if there is gender and tenure diversity like in in terms of like people's experience levels like they've been work longer with you with these two things you actually have people producing software that's better in less time and this is like you know a scientific study that you know you have to be able to claim like statistical significance and all that it's not just hand waving and there's a they've also done a study on people disengaging from open source so women disengage faster than men and they've also find that you know people actually you know as much as we claim that we don't really care about gender the first thing we see is gender and like how active you are as a developer like just you know like psychologically this is what we do so just be aware this is all real and there's some studies that you can look at I'm not gonna talk more about it I'm just gonna say like you know if you actually look in those look into those studies we have science that actually says that diversity and in mentoring people mentoring each other makes you build better software faster like like actually really not just like hand wavy like I'm a woman having opinions like this is serious you should really pay attention like it's science like don't shoot the messenger okay alright so I have a few slides so this is you know so I made a point about like we got all these new people coming in we should care about mentoring right and also that you know frameworks are kind of a thing that newcomers use so you know I've shown one or two of these I've shown this slide before basically and this is one of the first pieces of data I started collecting I've been following the things that company now called synopsis use be called black duck and tide lifts of and publishing and basically they survey companies and they've been doing this for a long time and in 2010 39 percent of 1200 companies or about a thousand companies that they surveyed said that the companies you know that they ran on open source so you know 39% 2010 that's not a lot by 2015 but almost eighty percent of those people said well yeah we run on open source everything that we do is based on open source software so this is like also you know like just somebody answering a survey how correct is this really I'll show you we have more data but the main point here is that you know this is up over to 2x since 2010 so basically in five years you see everybody kind of you know switching over to using open source stuff over over proprietary things I think this is all not a surprise we all in this room use open source software if you do anything in Big Data it's all open-source right so the point here is in since 2010 people started depending a lot more on open-source and there's a lot of reasons for this then the main reason in 2016 was that people thought the quality of solutions was was better and in 2015 people who come two-thirds of company said that they wouldn't even consider proprietary solutions so that's also interesting like everybody is sort of fixated on like we're just gonna use open source by 2015 so sorta became the default choice then in 2017 60% of companies said that they've increased their their usage and the reason why the main reason is that they don't want you know vendor lock-in but these are all surveys so we actually now have some data that's been published about you know sort of like anonymized scanned code bases right so synopsis scanned over 1100 commercial code bases and found that 96% of those code bases contained open source components and even more mind-blowing or actually maybe not mind-blowing if you live if you work in a big organization an average of 257 open source components they found per application it's like an insane number do you have any clue what any of that stuff is if there's any security vulnerabilities that's also scary for the record not my point though so 96% of companies used open source in 2018 and 2017 36% of the code base that they scanned was open source component so like I know I'm getting it just numbers and it's kind of hard to remember these numbers but like try to make like a timeline in your head so 2017 36% of the code bases that they scanned were open source components in 2018 that number increased to 57% this is crazy right so many applications now contain more open source code than proprietary code is what they found which is shocking so tide lifts also goes and does these sorts of things and they found something very similar so this was in 2018 they did basically they found that 70% of people's apps are essentially source components so there are most of people's apps but not their job to support take care of 20% of the applications that they were looking at we're actually the business logic that people wrote and then 10% was commodity infrastructure so things like kubernetes and docker and MongoDB stuff like that so that's absurd right like that's completely insane like let's just you know take this in for a second 36 percent of code bases were opens made you know was open-source components in 2017 according to synopsis and by 2018 that number became 57% and tide lifts said that that number was actually 70% so these two come in the same year I don't care like it fine we're off by 10 or 20 percent whatever that's still insane that's like a majority right so that's you know pretty shocking to think about especially given like you know we all know that there's open-source sustainability problems but I like holy crabs that much of our apps are actually things that what you really can't say very much about so these two points I'm going to make really quickly the the bottom line here is that you know this has been something that is not new we all know this is not new for a long time especially startup founders have been encouraging other people to just breedings out of open-source components and not send best effort re-implementing things are paying for proprietary solutions and so this is the co-founder of Instagram in 2011 basically saying you know borrow instead of building whenever possible that you can get all of your infrastructure essentially for free we all essentially do that now right and Nadia I ball in 2000 and I think I think this was 2015 she made the observation that I think that was a year that Instagram was acquired or maybe it was year before that I don't know but at the time that Instagram was required acquired she observed that probably that 143 million dollars of their 1 billion dollar acquisition was probably due to open source which is you know also kind of like an interesting perk right like things that they were used is worth about 143 million dollars uh-huh and yet at least according to tide lifts these are numbers they published for their 2018 professional open source survey 62% of respondents say that they are required to financially support the open source work with their own funds or that they receive no open source fund I'm sorry external funding at all so this is actually I think a good graph because it shows that a lot of the open source stuff that we work on are that we use nowadays at least half of it or almost half of it is supported by somebody like usually corporate subsidy an employer is basically paying for somebody to work on an open source component that other companies are using this is like I think the only way forward I don't I mean also people who do you know these donations and whatnot like it's been shown that donations don't really work because they're you know random amounts of money over periods of time that's not enough to sustain a human on so this is actually incur an encouraging number although 62% is also still scary that like if I look at 70% of people's applications being open-source components and I know that of that 70% 62% is some random people doing this you know on their weekends that's a little scary and and I'm just gonna make the point without going into it into too much detail but just make the point you know that of course there are things that are unsupported that we don't realize we all remember in 2014 two-thirds of all web at that time we're using open SSL and the heartbleed bug was discovered in that year and what was happening sort of on the person level is really interesting so there was two guys both named Steve or Steven Steve Marcus noticed that you know he was a contributor to open SSL he noticed that Steve Henson Steven Henson was working full-time on open SSL like and Markus was just you know like you know volunteering around his job right so Markus was like well hey hey Steven Henson what do you do for for a living and basically you know as a go so so Markus was a government contractor which you know usually they don't make tons of money and if he found out that at that time Martin Steven Henson was only making about 1/5 of what Steven Marcos was making which is very little and Markus at the time also considered himself to be a strong programmer but he thought that you know Steven Henson was really like a god amongst programmers and he had been working on open SSL since 1998 which is like you know if he's making it this meager wage since then that's pretty pretty insane he writes it always assumed as had the rest of the world that the OpenSSL team was large active and well resourced and reality OpenSSL wasn't even able to support one person's work because Steven Henson was doing contracting work around the side and you know we all know in retrospect we didn't know about any of this until it happened right so there's a bunch of projects out there that we don't know that these sorts of problems exist and I've shown this before I think it's interesting so this is a research group has basically worked out the truck factor of many popular open-source projects on github the main point here is that of the top 133 most active projects most of them have a truck factor of one or two which means if one or two people that we're working on the project disappeared then the project would essentially die because nobody could take it over and just to zoom in and show you some of the names this was a few years ago this is probably 2015 that these numbers came out so that maybe this is all changed by now but it's just interesting to note that you know a lot of these things don't have super high truck factors back in 2014 or 15 I guess and so you know that brings me to the community point right like how do you you know how do you increase truck factors well you should probably not fail at community an ecosystem then right because you need more people to work on this open source project with you and there's research results that point to that so in 2013 this is actually the best paper at Uppsala which is like the basically practical programming languages conference and they basically found after doing a bunch of but like tracking 10 years of open open source depository data and and doing us doing surveys on developers they found that they were trying to answer this question about what parts of the programming language influenced developer decision making and the most important thing is you know there are being you know open source libraries to use and that you know people can easily extend existing code which means that there are people who know the programming language already right and so what they observed they saw that you know these are all extrinsic factors not actually like features of the programming language these are all things about the community so they basically found that developers value open source libraries as a dominant factor in producing programming languages social factors not tied to intrinsic language features such as existing personal or team experience also rate highly so like this community factor turned out to be the most important thing for people adopting at least new programming languages is probably also true for frameworks so you know the point here is that you know you could really argue that the success of some open-source project is actually largely attributed to its community and the people who sort of rally around it this is arguably some of like one of the most important things because these people are fixing it these people are building new things on top of it these people are answering questions it doesn't matter how good the thing is and if these people aren't there to to make it better tide lifts all has data on this basically and the data says here that you know when they asked all these people in 2018 what was most important the most important things were ultimately extrinsic factors right related to community so reliability and maintenance the fact that people were regularly maintaining it was the most important and then the next important next most important which is you know almost the same at the same level was an active community so basically having an active community that was using and supporting it was one of the most important things for people making decisions to adopt software and so you know if you look at all of the other factors that they ask questions about respondents rated and active community as being over 20% more important than just the popularity of a project so that's hugely important right so and this is not just touchy-feely stuff this is not just like oh yeah we should love each other this is you know like life or death of the things that you use and make up large parts of your code base you know is that we have to be good at this so to stay alive there should be a relentless focus on growing the community an ecosystem and that's not easy and like not a lot of people know how to do that so you know again this is not like like let's give everybody hugs but you know community ecosystems are actually the most one of the most important things to an open source project staying alive so let's try to wrap all of these things up because I gave you a whole bunch of data points now you're probably like oh my god like this was too much so I had these three areas that I was I was I was talking about right so on the one how he built software at least it appears and what people say and what these surveys that have scanned thousands of codebase I'm sorry these these apps that have scanned thousands of codebase tell us is that at least now more than even five years ago three years ago we largely glued together a lot of open-source components instead of implementing them from scratch like this is just what we do now and of course there remain a lot of sustainability issues that we should at least be more cognizant of obviously I have strong opinions about how we can try to correct these things please talk to me about it but you know these the first two are kind of related right because you know if this one you know doesn't get better we have less nice frameworks that we can glue together to build apps with right and on top of that you know our idea of software developers is rapidly changing just given to this massive uptake or increase in the number of people that are doing these jobs now so most people are extreme newcomers and like you know if you have a few years of experience you have no idea how many people are out there and completely new to this field and we have two in the next couple of years somehow uptake those people and I mean you know I don't care how good you are at some weird concurrency thing like if you don't get better at mentoring this is gonna be bad for all of us so some of the takeaways like what do we what do we actually do so as you can see I'm just gonna keep on repeating the phrase we're getting a lot of newcomers like and I'm gonna wave my hands again about that um and also that you know those newcomers don't necessarily all have fancy computer science degrees maybe they don't actually need one because you know the jobs nowadays are often gluing together frameworks frameworks are now king I mean at least if you believe the the numbers that these other companies are publishing you know people are gluing together open-source components and 20% of the applications that people are building now are actually software that you have to write and the other 80% is reuse stuff right so maybe you don't need a fancy CS degree maybe that's fine maybe we should be okay with that and as a professor I should probably not say that but whatever another cool thing like you know how we've at least in an academia land you know people have been saying since like the 80s or the seventies that like we should try to be good at reusing software and I think that if there's a good thing to be said here we're now really good at it we really actually reuse things a lot but the things that we were using we're reusing are all largely open-source components and you know corporate subsidies having helping about half of these open-source projects that we're gluing pieces together I mean again given these these statistics so you know 60% is is not funded right and people are funding it themselves that's important we should be aware of this we don't want more heartbleed situations right and again I'm just gonna point out that we're getting a lot of newcomers so these things were all related like I said this is all all the same problem I just don't know how to to say it more succinctly basically newcomers depend on new open source projects and they they depend on it being easy to use and that's how we update or that's how we bring people into our field you know and applications are becoming a majority open source which is cool I guess right but you know we continue to struggle with sustainability and open source and we have to somehow address this so if there's something that you should like walk away with other than just being like oh Heather give me a bunch of alarming statistics you know I have a few points that I think are important for different groups of people I presume there's like no educators in the room like three and like everybody else is either an engineer or a manager for managers I think that it's important that we all start embracing the need to be better mentors and to actually put a greater focus our emphasis on mentorship sooner than later and also you know it's really important that we figure out how to give back to the open source software that we depend on and practicing engineers we can repeat it again like you should actually try to get good at mentorship I have a resource out there that's kind of nice that you know people can look if you'd like I mean I didn't write it I found it it's nice and as educators I mean like I said I probably like one of the only people in this room that's currently doing like CS education in a university setting but maybe we should we should you know be cool with not everybody needing a CS degree all the time and maybe we should try to instead figure out how to more quickly and and affordably help people get the skills they need as fast as they can so that they can just become an application developer and fill these needs that we have in our society and again we should not forget to actually give back to open source and invest in the stuff that we build on it's not enough to just throw money at a project I had all these community slides the whole point is that we actually need to invest time and I know gene it's community and this is the I'm just gonna let you like this is there's a link here and the slides were posted but this is a really good men good resource on mentoring which actually just came out a few weeks ago which is really nice and at that point I'm happy to answer questions but I think I can do it like when we're changing tracks the slides are posted here and I'm going to give them as well to uh to Alexian and co so in case you want any of these statistics you can just like like look at the slides and with that thanks for listening and you want to come talk to me come on up [Applause] [Music]