data.bythebay.io: Josh Wills, Why The Best Minds Of Our Generation Are Thinking About Click Ads
how's it going everybody thanks for coming to see me I think the abstract for this talk was something to the effect of like it'll be Josh Wills talking about stuff with no additional substance or whatever so I guess thank you all for coming I suppose for that or for those of you who just basically were sitting in the room and forgot to leave for a morning interesting talk I'm sorry I apologize um a little bit about me uh I am currently Slack's head of data engineering uh I've been in slack for about six or seven months now which I think makes me like an Oldtimer in in sort of slack years since the whole company's like two years old uh I worked at Cloud era for four years before that uh as was director of data science there uh I used to work at Google for a while and I ran data science teams and data engineering teams and all that kind of good stuff uh I wrote a tweet once about data scientists that you've is certainly seen and that I never want to talk about ever again um and my first job at Google was uh advertising so like when you ever did a search on Google like 20072 2008 sort of time frame and ads showed up that was me working on the ad system working on the auction figuring out where I should go how much people should pay all that kind of good stuff you're welcome nobody ever thanks me for that um ever since then I've been trying to do work in order to make a up for my couple years spent as an Advertiser sort of like a Penance or a Karma or something like that with varying degrees of success so um I worked at cladera for a guy named Jeff hammerbacher who has this famous quote uh the best minds of my generation are thinking about how to make people click on ads that sucks and that might it's probably another quote that you've seen if you've done any stuff in data for any period amount of time um Jeff went on the Charlie Rose show once and said that that quote will be on his tombstone and he's probably right that's probably going to be like his most lasting contribution to western civilization right is roughly that quote um and I have like some sort of personal experience with the quote because uh I was I don't know so it's not really a way to it's not really a humble way there's there's no Hallmark card for saying I am one of the best minds of my generation um I'm not one of the best minds of my generation I'm like the guy who hangs out in the room with the best minds of Our Generation and like brings them coffee and sandwiches and stuff like that so I'm I'm not in the club but I'm like sort of next to it um and I was working in Google for a long time and Jeff was trying to recruit me to cladera and you know basically his his ultimate technique was to get me to realize that at the end of the day anything you're doing at Google or Facebook is fundamentally devoted to uh advertising like that's that's sort of the point of it right no matter what you're doing there's actually a great quote I just saw on Twitter like literally five seconds ago and I'm going to read it because it was really funny um Google Google's doing IO right now right and this gu this guy tweeted Google products are like the massages given to Kobe beef they're not for the benefit of the cow they're to make the cow a better product yeah think about that one for a second right but that's actually like pretty goddamn good right I'm I'm going to ahead and retweet that right now um yeah there's some truth to that y'all like not for nothing um there's some truth to that so anyway the way Jeff got me to go work in Cloud era was to get me to realize that uh at the end of the day I really personally don't care whether Google or Facebook is the dominant advertising force in the world like I really don't like I mean who like honestly who cares right I guess like maybe you know Larry and Sergey and Mark maybe those guys care because it's like a billion here and a billion there or whatever but like I personally I don't really care um I wanted to go work at Cloud era because I wanted to go work on other stuff I wanted to go work on Health Care stuff and oil and gas stuff and climate change stuff and there's like lots and lots and lots of other stuff that we can do with with all of this great great data technology that's been invented over the past decade or so um but as I've spent like the last I guess six years now like like sort of removed from adte and not doing adte anymore I would be lying if if I said I didn't miss it a little bit um and I want to kind of talk about why for a bit I've been thinking about this lately like working at slack um why did I get an advertising in the first place well it's because there's honestly a lot of money in advertising it's like why did people do you know like subprime mortgages and stuff like that and work at hedge funds it's because there's like a lot of money there um and there was a lot of other sort of great stuff that came along with that money um and then one of the things was just like really really really smart people working on really really interesting hard problems um and sort of putting their Solutions out there and like releasing them in as open- Source software that anyone who was basically as smart as they were and capable of running the stuff could use to work on any problem that they were interested in right so all of the stuff around Hadoop all of the stuff around Kafka really all of like basically all of the nice stuff we have is through the largess of these very very profitable advertising companies that just gave all this stuff away um largely as a technique to recruit other software Engineers more or less to come work on on their awesome stuff um so I you know I don't want to say that like advertising is purely a bad thing it's it's not um it's created a lot of lot of very very cool stuff and solved a lot of really interesting computer science problems over the last 10 years in a way that is sort of open and available to everyone and that's that's a very worthy thing um and sort of the other thing that I I I loved and is maybe like talked about a little bit less I loved the culture of working on really really large scale machine learning problems um I loved working at a place where like the stuff that I did was really the center of the universe it was like the center of what the company did in some sense and everyone was really align LED around data infrastructure like really you know being sort of true to ourselves and being like like more so than anywhere else I've ever seen actually like following the scientific method right and really really running experiments and not relying on the sort of the highest paid person in the room um realizing that everybody was in this together having like world-renowned machine learning researchers carrying pagers because they were on call for Ops Duty support just the culture of working in advertising doing like large scale adte like just basically big ass logistic regression models is actually really really great um I am still like really good friends with most of my my you know uh peers from my Google days just because that experience of working together on something like that was super fun and just in many ways probably like the best working experience in my life um including all the stuff I've done at cladera and honestly including all the stuff I've done at slack I've never had that sort of I I have a friend who's who's a Michelin star chef and she says that like working in a really great kitchen is like that it's hard um it's high pressure but everyone is working towards the same thing and everyone is really really good at what they do and it is a phenomenal working experience if you ever have the chance to do it I I highly recommend it um but at the same time you know I guess I'm a little worried about I I don't know to quite say this um we've gotten really really really good at building machine learning models for ad Tech stuff um really really good at it and in fact we've gotten so good at it that there aren't really any people involved in advertising anymore um at least to the extent to which they people are involved in sort of at best a tangential sense and what I mean by that is there are algorithms that are so you know so complicated that no one can understand them deciding what ads to show um there are algorithms on the other side like in botnets and stuff like that that are clicking on the app ads that are so complicated that no one can understand there are other machine learning algorithms on the other side that are trying to defeat the Bots and there I mean it's just basically like it's it's just essentially it's an army of machines battling out over this world that like a human occasionally like stumbles into the killing field sometimes and like clicks on an ad but there's just not there's no people there it's it's just all machines it's all computers um yeah and I don't know about y'all I mean I'm like I'm I'm pretty introverted despite what I'm doing right now um but I you know I basically like people I don't know it's like one of those things like one of the things I like about slack is there there are people there right there are people Land There are Bots um but there are definitely people uh I'm reasonably sure yeah um so yeah it's I I guess it kind of feels to me in the same way that I think maybe people who've been doing hedge fund stuff for a long time it can feel just sort of empty and draining and just abstracted away from anything that matters at like a human scale um in a way that is just ultimately it's ultimately like not nourishing it's it's like eating a bunch of candy or junk food or something like that it feels very rewarding at the time see the thing about the the picture of me at my house swimming around in my my big giant you know uh whatever Warehouse full of money um but it's it's ultimately not like good for you uh as as like a person basically yeah I wish I made that a little bit crisper I'm not going to lie but let's let's move on um so I've left advertis in now uh and I've left it for quite some time uh and I'm I'm sort of out you know on my own in the sort of lame as a Rob sense of the term um and I sort of like Miss advertising and miss I miss parts of the camaraderie I miss aspects of I miss the sort of the sheer awesomeness of my colleagues and stuff like that um but I do generally feel better about the work I do every single day it is like more rewarding and more meaningful to me um and I think the thing that I really miss the most the thing I miss the most is the camaraderie the thing I miss the most is the relationships it is it is the sense of everyone being in this together and it's the thing that I feel like it's it's not it doesn't exist in sort of the non the nonata world like it doesn't really exist yet at slack I don't think it exists at most of the companies in the world that are doing sort of non-ad tech stuff and I want to talk about that a little bit because in order for data to really really truly take over then we need the best people in the world to not be thinking about clicking on ads um we need them to be thinking about these problems and the way to get them to think about these problems is not just about technology and it's not just about about money it's about the culture and it's about the experience of working at these places um that we need to talk about and we need to solve and so let's let's I'm not going to solve the problem today obviously this is just a talk but I do want to talk about my experiences and roughly like how I think about these things um I thought the slide would Advance there it goes so let's talk about analytics engineering that's kind of where things start here um I kind of feel like you know in a lot of ways I I think like for my next career after I leave slack I'd like to become a therapist for data people roughly speaking that's actually what I be like oh thank you okay that's a few I I don't know I I kind of feel like I played this role in a sort of ad hoc sense now um I have like you know basically friends who are the first data analyst or the First Data engineer working in a company and they reach out to me because they're they're basically having a horror time and they're very unhappy and they think there's something wrong with them they think that like they are um stupid or that they should be like way better at the stuff or other companies are more data driven than they are or something like that um and it's really not true it's it's horrible for everyone um it's not you're not alone here and I feel like a lot of my my sort of the the the value I provide is just like telling people that right um and I feel like in some ways though even at slack where we have like a devoted Analytics team and a devoted data engineering team in some ways like we're not alone anymore there's not just one person there's like a there's a team there but we still have like a bunch of problems we still have a bunch of like sort of cultural sort of riffs um that we have not solved yet and that we need to talk about and need to figure out how to solve and again we're a little bit bigger and a little bit more successful but we haven't solved these problems yet so even you know even the sort of larger teams need some kind of group therapy or couples counseling or something like that in order order to be able to work together better um and I think you know where it starts is trying to figure out like what exactly is it we both want out of our relationship um what do the analysts want what do the engineers want what do the data scientists want what do the data Engineers want um in so far as I can tell uh data scientists want to build really really cool models and really really cool metrics and dashboards that fundamentally change the way people think about a problem or like either the executive level or whatever and then go talk about them at conferences so they seem really really cool to all the other data analysts and data scientists and stuff like that um data Engineers on the other hand want to build like truly awesomely scalable very very reliable systems that handle all failures perfectly and deliver data exactly on time and are super awesome and fantastically amazing and then they can go at conferences and talk about how awesome they are at scaling and and reliability and all that kind of good stuff um so fundamentally we both want to go to like sort of our separate conferences and talk about how awesome we are at our separate things um but the reality is these these two goals are like almost always in conflict with each other um building things that are reliable and scalable means that they're probably going to be fairly simple and not necessarily that interesting um at least from a data scientist perspective and doing all of the sort of crazy that a data scientist wants to do in like r or python is not going to satisfy the reliability or scalability needs of a data engineer so when we're not all working towards the same sort of productionize machine learning kind of problems we end up wanting different things and that is like sort of the core of our relationship problem we want different things out of our relationship um and really the problem is not just data engineering and data science this is this is maybe like hopefully my my other major contribution to western civilization will be talking about the infinite Loop of sadness um and it's really it's it's all about the relationship between um Ops data engineering data science and the business and essentially um unreasonable requests flow around the loop uh the Loop of sadness so the business um has no understanding whatsoever of what is and what is not possible with data so they make ridiculous requests of the data science team the data science team wants to run like basically terribly unscalable algorithms on like brand new hardware on untested versions of Hive and Presto and they basically Force the data engineering team to deploy this garbage into their clusters um so that they can use whatever shiny new feature that they are super excited about um the data engineering team is making insane requests of Ops in order to build their super ridiculously scalable storm pipelines that could have been easily solved by like a kofka consumer that was a bash script um and the Ops team goes around to the business and asks them for a ton of money because they need it to support all of this crazy stuff that everyone else is trying to do and the cycle basically spins out of control it just kind of keeps going like that because then the business is like well we're spending all this money we ought to be getting all this ridiculously cool analytics that we want blah blah blah blah blah and you can kind of see how this sort of goes out of control um I don't really know how to solve this problem but again let's just start by talking about it right this is this is the nature of the relationship between all these different parts of the organization um and as a result of this I think we end up being like Alone Together like we're sort of next to each other we're living our lives but we're essentially everyone in this like like basically everyone in this Loop is constantly trying to Outsource the adjacent people in the loop like more or less as far as I can tell the data Engineers want to automatically pump data into A system that automates what the analysts do the analysts want to you know so on and so forth okay good deal um so what do we do about this what do we do about this like what what is the solution to this problem again I don't know I'm probably going to keep saying I don't know is like a general theme of this talk um but let's start with first principles all right and the first principles are at least for me at least recently working at slack in order to build data infrastructure you only need to understand and appreciate two things Apache Kafka and fron Kafka and Apache Kafka is obviously important it's like oxygen for data people um but nonetheless the the France Kafka part is even more important um I'm going to quote sort of a little bit of David Foster Wallace here he wrote this great essay about kafka's sense of humor it's in a book called consider the lobster if you want to read it um and basically he's saying like students can't understand kafka's sense of humor um they can't appreciate the really Central Kafka joke which is that the horrible struggle to establish a human self results in a self whose humanity is inseparable from from that horrific struggle and the true the same is basically true of data infrastructure um the horrific struggle to establish data infrastructure results in an data infrastructure that is inseparable from that horrific struggle it's it's horrific for everyone it's absolutely terrible sort of like bubble gum and you know toothpick sort of like jamed together disaster um for absolutely everyone you're not alone here and and that that disaster is our home like that is that is where we belong it's not something we can ever really Escape it's it's what we're here to do more or less so instead of having an infinite Loop of sadness we really need to have an infinite Loop of empathy and in particular needs to be an infinite Loop of empathy between data engineering and data science because we have similar relationships like like the data engineer should understand that from the analyst perspective they are like the Ops Team and the data scientist should know that like from the data Engineers perspective they look like the business team because they effectively are right and if you bring that kind of empathy to sort of think through what is is this like for the other person based on your own experiences I think it becomes easier for us to work together now like I said um machine learning if you can do sort of production scale serious you know large scale machine learning it's relatively easy to get everyone on the same page but a lot of companies don't really need to be doing large scale production machine learning like it's not really necessary um but everybody needs to do ETL everyone needs to extract data from somewhere perform some Transformations on it and load it somewhere else this is an absolutely brilliant comic by the way and the fact that more of you aren't laughing is like making me judge everyone of the audience right now but like go back and watch this later um everyone does ETL and the way that you do ETL to me says a lot about the data culture inside of your organization so um the first option I've seen I'm this is sort of like Facebook Facebook SQL Centric ETL um at Facebook every ETL pipeline is basically a hive job Hive or Presto for absolutely everything it is a very SQL Centric kind of ETL that any analyst or data scientist can do and data engineers at Facebook are assigned in kind of a relation there's roughly um one data engineer for every two data analysts roughly speaking and the data engineer is essentially the data analyst slave um they write udfs they optimize pipelines they're basically like the servant of whatever crazy goddamn thing the data analyst wants to do um and this is like as a sort of data engineering person like not a healthy or good relationship I don't like Advocate this relationship but this is fundamentally what happens when data analysts are kind of fundamentally running the show when it comes to doing data at your company um option two I'm going to call out Twitter for this one Twitter is the land of the jvm Centric ETL where in order to write an ETL pipeline at Twitter you better have a sort of advanced degree in abstract algebra and know your sort of monoids and rings and principal ideal domains really really well um essentially like at Twitter uh the core ETL pipelines are completely automated completely written in Scala uh and basically maintained by a high priesthood of Engineers who occasionally Dain to let the data analysts like look at some slice of the data if they feel like it now again this is actually much more appealing to me as a data engineer but I don't necessarily think of this as like the optimal State of Affairs either okay so what I want to come up with at slack is a third way a third way that makes this really not the ETL thing nothing but this okay fine um a third way that b basically makes everybody equally unhappy that's that's what a Third Way means a line break after every sentence I mean that's just the worst anyway what's what is the third way look like um there's a bunch of different sort of attributes of like data stuff now that are sort of informing the way that I want to think about this Third Way um the first and foremost is the rise of spark spark is if data is the new oil spark is the new standard oil spark has its tentacles into absolutely everything spark is completely unavoidable if you're doing any kind of data stuff at least in San Francisco um these days all right um thing two there are way too many goddamn streaming engines right now um Googled that we had this great joke um there were two ways to do anything the way that was deprecated and the way that didn't work yet um batch processing is deprecated and stream processing doesn't work yet that's the problem we have none of these things actually work aside from Kafka like Kafka is the only thing that works um spark streaming oh my God like what a dumpster fire all right it's just the absolute worst okay like I just how who it's like who I want to slap them like who taught you to manage memory for the love of God okay anyway leaving leaving that aside um there are too many streaming engines right now I have to pick one I need to deploy a streaming engine to serve some analytics to spark platform developers or sorry to slack platform developers that's Sparks again the tentacles are everywhere um slack platform developers I don't know which one to choose um whichever one I choose will be wrong that's all I know okay there's too many of these things um that said the nice folks um at Google like the cloud data flow team and the Apache beam Community have put out some really really really nice design patterns and some really very cool new ideas for thinking about how to handle like late arriving data and like real world data pipelines that resonate with a lot of my friends who are like thinking about these problems and have been experiencing them lately like making windows and time a first class primitive in your data pipeline abstractions is basically a really goddamn good idea and something we should have thought of a long time ago so even though all of these engines are terrible we roughly know sort of in a broad sense what patterns are right for solving this problem all right and last thing um the real problem of data engineering of data science is change that is the real problem it's not scale scale is not a real problem anymore it's not a real problem at slack it's honestly not a real problem at anywhere besides like Google and Facebook if you are not Google or Facebook use the thing that Google and Facebook open sourced it will solve your scale problem okay the real problem is change how do we handle Evolution how do we handle backfills how do we handle recovery that's the real problem of data engineering that is not super sexy to talk about at conferences but is actually the thing we need to solve and so with that I think I have like 30 seconds left but that's fun um I'm taking some inspiration from Deep learning I can just go okay cool all right I have some inspiration yeah talk that starts at two 30 it's in the female room but I just I'm going to break the rules and let him talk if you don't mind all right thanks sorry I won't hurt my feelings if any has to leave it's okay um as I think about solving this problem solving these problems building data infrastructure at slack I take some inspiration from Deep learning um between like torch and theano and tensor flow we have this pattern of a highlevel scripting language like be it in python or Lua or whatever um that is supports and integrated with a very low-level High Performance Engine in C C++ using gpus and all kinds of crazy stuff in order to do the really really hardcore processing at the core of these deep learning models and that's the kind of pattern I'm into for for data engineering as well I think we need a highlevel very very flexible scripting language for doing just the absolute brain dead stuff we have to do all the time and a low-level engine for doing the high performance aggregation stuff that we do all the time that we know how to optimize that we know we need to run really really efficiently um the other thing that's interesting to me like these days and in sort of programming is this evolution of like not quite static typing and this is like the flow language on top of JavaScript or hack that Facebook released this idea of like evolvable sort of typing where you can kind of go from like a completely a dynamically typed environment to a completely statically typed environment on your own pace whenever you feel like and while for a long time I've been a major advocate of uh true static typing for data pipelines I feel that now with spark with being being able to load data sets in memory and cache them and work with them interactively I've become much less obsessed about static typing it just the cost of like a type error has gone down to the point where like I I appreciate the flexibility and speed of a sort of dynamically typed language when I'm doing interactive analysis and then even better when I want to productionize that and turn that in ETL the ability to like add types to it um seems like a really really good solution to me and seems like a great way of approaching the problem um the other thing is that you know I work at slack and Once Upon a Time slack was this really cool game called glitch um and glitch was a game engine and the game engine was written in Java and the game engine had a level designer that was written in JavaScript so as it happens at slack I had a really really good Java Engine That Could support scripting from JavaScript already available to me and the result of this is something that you're probably going to be horrified by but that's okay um I am basically writing a sort of ETL description language entirely in JavaScript running on top of a scolar runtime engine no one's thrown anything at me that's awesome okay that's great um the way this work yeah exactly right everyone's too stunned and horrified to say anything um the way this works is you declare tables and you declare schemas for those tables that have column names and types and aggregations and all that kind of good stuff and that's executed in Scala Scala parses the JavaScript via the um via the nashorn engine turns it into some objects and does the actual execution but then of like your flat map function can be done written entirely in JavaScript it can be tested locally it can be tested in a browser whatever you want and then deployed out onto the cluster as part of like an overall sort of large scale pipeline the scol engine takes care of the orchestration wiring everything together and making sure that everything runs in the correct order the cool thing about this for my analysts is they don't really have to write udfs anymore they don't have to delve down to the internals of Hive and Presto they can just like write their scripting functions directly like things they would do as udfs can be done much cleaner in a much more testable way immediately in JavaScript everyone hates JavaScript and that's great because it's like if you can't program JavaScript you basically can't program does that make sense like every programmer hates JavaScript but because they hate it they have to know how to do it because otherwise okay never mind it's fine take my word for it it's pretty cool all right C cka exactly um additionally though because of the magic of spark SQL we can still incorporate SQL into this sort of flow whenever we need to so if I need a lookup table or if I need to do a join as part of the output to a pipeline I can always fall back to SQL and have spark SQL do all of the heavy lifting for me which is fantastic and so yeah this is roughly speaking the Brave New World I'm trying to create I want to create a meaningful collaboration between data analysts and data scientists on something fun that satisfies both of our needs it satisfies our need for realtime large scale streaming pipelines that satisfies their need for flexibility for exploration for all the kind of good cool stuff they get from data science um and then we can all sort of live together in a harmonious you know like magical Tower whatever this thing is like that's where we're going to live together it's like a biome or something whatever and uh yeah with that we're hiring and if you'd like to hack me help me hack on this stuff we'd love to have you at slack thanks very [Applause] much thank you very much J two minutes to spare tons of time tons of time um I'm hoping there are lots of questions please please okay are there questions question if there questions over there I'm not going to be able to see you so just as an FYI oh I'm I'm hanging it so oh okay cool all right you can see though because you're down there yes how's it going great Josh any uh tips on how to do the metadata management I noticed you sort of a hardcoding some table definitions in there and how do you manage the simple a task of the the sharing of that information oh so I think for us at least it's slack that like a lot of other sort of cloud native companies that we have a centralized Hive metastore that is our like single source of Truth for everything the definitions in like external Thrift files or in like these JavaScript files basically automatically propagate they whenever we're kicking off a job we compare the output table definition from the JavaScript to the existing table definition in Hive and if there's going to be new columns we just automatically add the new columns and if you add columns in a way that is incompatible with the existing schema we fail the job and we don't let you run any further but I think for a lot of us you know it's not perfect but I think the hive metast stor has become the closest thing to a centralized metadata manager that is practical and fairly useful and all that kind of good stuff so yeah anyway good talk lots of things to go with but how about this one you seem pretty um uh calm and attached with the Apache uh collection of tools and that's not the only game in town no but it's and and also a sub question of that is that uh spark squel is pretty relatively new yeah relatively new and it's very Central and unquestioned in your talk there yeah it's actually like not that terrible I guess sort of like the thing it's spark sequel has been around for a couple of years now and it sort of yeah no it has so I'm sorry shark has been around for a couple of years now right the thing that runs like Hive on top of spark and that's basically the way we use it like that's sort of it's if you want to run an inline Hive query I mean yeah like the Catalyst Optimizer gets things wrong sometimes and tungsten's kind of a pain you need to turn it off but like it basically works and you know the sort of interaction like really nice thing is having the clean interaction between like you know I fault spark for lots of things I will never run spark streaming like Over My Dead Body um but for batch processing and for doing kind of the clean integration of SQL and sort of arbitrary Scala or arbitrary whatever ever it's actually pretty good at what it does it's it's there's a lot of stuff that's like just a little off you have to hack around and that's annoying but like it's basically pretty good at what it does so so that's me being nice to the spark developers that's what passes for me being nice anyway yes you mentioned that uh it's either technolog is either deprecated or not working at all where does slack kind of fit in that Continuum oh that is a great question um email is deprecated spark slack doesn't work yet yeah no I almost like a lot of times I feel like like you know you go around San Francisco people tell you how much they love slack and I'm like [Laughter] really why like you live you live on this side of the fence it's like God this product is terrible like oh my God I can't believe we charge people money for this this is horrible we need like a ton of work to do to make this to make slack not terrible all right cool ripping on my own product thank you very much Josh thanks [Applause] everybody